Asterism: Pegasus and dispel4py hybrid workflows for data-intensive science

Filgueira, Rosa; Ferreira da Silva, Rafael; Krause, Amrey; Deelman, Ewa; Atkinson, Malcolm. 2017 Asterism: Pegasus and dispel4py hybrid workflows for data-intensive science. In: DataCloud 16, Utah, USA, 13-18 Nov 2016. Association for Computing Machinery.

Before downloading, please read NORA policies.

Preview

Text asterism-pegasus-dispel4py(23).pdf - Accepted Version Download (2MB) | Preview

Abstract/Summary

We present Asterism, an open source data-intensive framework, which combines the strengths of traditional workflow management systems with new parallel stream-based dataflow systems to run data-intensive applications across multiple heterogeneous resources, without users having to: re-formulate their methods according to different enactment engines; manage the data distribution across systems; parallelize their methods; co-place and schedule their methods with computing resources; and store and transfer large/small volumes of data. We also present the Data-Intensive workflows as a Service (DIaaS) model, which enables easy data-intensive workflow composition and deployment on clouds using containers. The feasibility of Asterism and DIaaS model have been evaluated using a real domain application on the NSF-Chameleon cloud. Experimental results shows how Asterism successfully and efficiently exploits combinations of diverse computational platforms, whereas DIaaS delivers specialized software to execute data-intensive applications in a scalable, efficient, and robust way reducing the engineering time and computational cost.

Item Type:	Publication - Conference Item (Paper)
Date made live:	07 Apr 2017 15:04 +0 (UTC)
URI:	https://nora.nerc.ac.uk/id/eprint/516823

Actions (login required)

View Item

Document Downloads

More statistics for this item...