Scripting the cloud with skywriting
Derek Gordon Murray, Steven Hand · 2010
Recent distributed computing frameworks—such as MapReduce, Hadoop and Dryad—have made it simple to exploit multiple machines in a compute cloud. However, these frameworks use coordination languages that are insufficiently expressive for many classes of computation, including iterative and recursive algorithms. To address this problem, and generalise previous approaches, we introduce Skywriting: a Turing-powerful, purely-functional script language for describing distributed computations. In this paper, we introduce the main features of Skywriting, and outline our novel cooperative task farming execution engine.
Cited by 27 works.
Authors
References
- The anatomy of a large-scale hypertextual Web search engine
- The Google file system
- MapReduce: Simplified Data Processing on Large Cluster
- Dryad
- Pig latin
- Distributed computing in practice: the Condor experience
- Spark: cluster computing with working sets
- Interpreting the Data: Parallel Analysis with Sawzall
- MULTILISP: a language for concurrent symbolic computation
- DryadLINQ: a system for general-purpose distributed data-parallel computing using a high-level language