Asterism: Pegasus and dispel4py hybrid workflows for data-intensive science

Filgueira, Rosa; Ferreira da Silva, Rafael; Krause, Amrey; Deelman, Ewa; Atkinson, Malcolm. 2017 Asterism: Pegasus and dispel4py hybrid workflows for data-intensive science. In: DataCloud 16, Utah, USA, 13-18 Nov 2016. Association for Computing Machinery.

Before downloading, please read NORA policies.

Preview

Text
asterism-pegasus-dispel4py(23).pdf - Accepted Version
Download (2MB) | Preview

Official URL: http://dl.acm.org/citation.cfm?id=3018101

Abstract/Summary

We present Asterism, an open source data-intensive framework, which combines the strengths of traditional workflow management systems with new parallel stream-based dataflow systems to run data-intensive applications across multiple heterogeneous resources, without users having to: re-formulate their methods according to different enactment engines; manage the data distribution across systems; parallelize their methods; co-place and schedule their methods with computing resources; and store and transfer large/small volumes of data. We also present the Data-Intensive workflows as a Service (DIaaS) model, which enables easy data-intensive workflow composition and deployment on clouds using containers. The feasibility of Asterism and DIaaS model have been evaluated using a real domain application on the NSF-Chameleon cloud. Experimental results shows how Asterism successfully and efficiently exploits combinations of diverse computational platforms, whereas DIaaS delivers specialized software to execute data-intensive applications in a scalable, efficient, and robust way reducing the engineering time and computational cost.