FireWorks: a dynamic workflow system designed for high‐throughput applications
Anubhav Jain, Shyue Ping Ong, Wei Chen et al. · Concurrency and Computation Practice and Experience · 2015
Summary This paper introduces FireWorks, a workflow software for running high‐throughput calculation workflows at supercomputing centers. FireWorks has been used to complete over 50 million CPU‐hours worth of computational chemistry and materials science calculations at the National Energy Research Supercomputing Center. It has been designed to serve the demanding high‐throughput computing needs of these applications, with extensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) provenance and reporting for long‐running projects, (iv) automated duplicate detection, and (v) dynamic workflows (i.e., modifying the workflow graph during runtime). We have found that these features are highly relevant to enabling modern data‐driven and high‐throughput science applications, and we discuss our implementation strategy that rests on Python and NoSQL databases (MongoDB). Finally, we present performance data and limitations of our approach along with planned future work. Copyright © 2015 John Wiley & Sons, Ltd.
Cited by 547 works. doi:10.1002/cpe.3505 Open-access PDF
Authors
- Anubhav Jain
- Shyue Ping Ong
- Wei Chen
- Bharat Medasani
- Xiaohui Qu
- Michael Kocher
- Miriam Brafman
- Guido Petretto
- Gian‐Marco Rignanese
- Geoffroy Hautier
- Dan Gunter
- Kristin Aslaug Persson
References
- Architectural styles and the design of network-based software architectures
- IPython: A System for Interactive Scientific Computing
- Globus: a Metacomputing Infrastructure Toolkit
- Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences
- Distributed computing in practice: the Condor experience
- Python for Scientific Computing
- Commentary: The Materials Project: A materials genome approach to accelerating materials innovation
- Exploring Network Structure, Dynamics, and Function using NetworkX
- Taverna: a tool for the composition and enactment of bioinformatics workflows
- Scientific workflow management and the Kepler system