Increasing the Throughput of Pipe-and-Filter Architectures by Integrating the Task Farm Parallelization Pattern

Abstract

The Pipe-and-Filter style represents a well-known family of component-based architectures. By executing each filter on a dedicated processing unit, it is also possible to leverage contemporary distributed systems and multi-core systems for a high throughput. However, this simple parallelization approach is not very effective when (1) the workload is uneven distributed over all filters and when (2) the number of available processing units exceeds the number of filters. In this paper, we explain how we utilize the task farm parallelization pattern in order to increase the throughput of Pipe-and-Filter architectures. Furthermore, we describe an associated modular self- adaptive mechanism which enables the automatic resource-efficient reaction on unevenly distributed workload. Finally, we refer to an extensive experimental evaluation of our self-adaptive task farm performed by us. The results show that our task farm (1) increases the overall throughput and (2) scales well according to the current workload.