Incremental, iterative data processing with timely dataflow
Derek Gordon Murray, Frank McSherry, Michael Isard et al. · Communications of the ACM · 2016
We describe the timely dataflow model for distributed computation and its implementation in the Naiad system. The model supports stateful iterative and incremental computations. It enables both low-latency stream processing and high-throughput batch processing, using a new approach to coordination that combines asynchronous and fine-grained synchronous execution. We describe two of the programming frameworks built on Naiad: GraphLINQ for parallel graph processing, and differential dataflow for nested iterative and incremental computations. We show that a general-purpose system can achieve performance that matches, and sometimes exceeds, that of specialized systems.
Cited by 36 works. doi:10.1145/2983551 Open-access PDF
Authors
References
- MapReduce
- Dryad
- Bigtable: a distributed storage system for structured data
- Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
- Parallel database systems
- Synchronous data flow
- DryadLINQ: a system for general-purpose distributed data-parallel computing using a high-level language
- Scaling distributed machine learning with the parameter server
- Dremel
- PowerGraph: distributed graph-parallel computation on natural graphs