CIEL: a universal execution engine for distributed data-flow computing
Derek Gordon Murray, Malte Schwarzkopf, Christopher Smowton et al. · 2011
This paper introduces CIEL, a universal execution engine for distributed data-flow programs. Like previous execution engines, CIEL masks the complexity of distributed programming. Unlike those systems, a CIEL job can make data-dependent control-flow decisions, which enables it to compute iterative and recursive algorithms. We have also developed Skywriting, a Turingcomplete scripting language that runs directly on CIEL. The execution engine provides transparent fault tolerance and distribution to Skywriting scripts and highperformance code written in other programming languages. We have deployed CIEL on a cloud computing platform, and demonstrate that it achieves scalable performance for both iterative and non-iterative algorithms. 1
Cited by 225 works. doi:10.5555/1972457.1972470
Authors
References
- Identification of common molecular subsequences
- MapReduce
- The Google file system
- Option pricing: A simplified approach
- Dryad
- Can programming be liberated from the von Neumann style?
- MPI: A Message-Passing Interface Standard
- Pig latin
- Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
- Spark: cluster computing with working sets