Spark: cluster computing with working sets
Matei Zaharia, Mosharaf Chowdhury, Michael J. M. Franklin et al. · 2010
MapReduce and its variants have been highly successful in implementing large-scale data-intensive applications on commodity clusters. However, most of these systems are built around an acyclic data flow model that is not suitable for other popular applications. This paper focuses on one such class of applications: those that reuse a working set of data across multiple parallel operations. This includes many iterative machine learning algorithms, as well as interactive data analysis tools. We propose a new framework called Spark that supports these applications while retaining the scalability and fault tolerance of MapReduce. To achieve these goals, Spark introduces an abstraction called resilient distributed datasets (RDDs). An RDD is a read-only collection of objects partitioned across a set of machines that can be rebuilt if a partition is lost. Spark can outperform Hadoop by 10x in iterative machine learning jobs, and can be used to interactively query a 39 GB dataset with sub-second response time. 1
Cited by 4,079 works.
Authors
References
- MapReduce
- IPython: A System for Interactive Scientific Computing
- Dryad
- Map-Reduce for Machine Learning on Multicore
- Pig latin
- Generative communication in Linda
- Map-reduce-merge
- Mesos: a platform for fine-grained resource sharing in the data center
- DryadLINQ: a system for general-purpose distributed data-parallel computing using a high-level language
- Delay scheduling