Incremental, iterative data processing with timely dataflow

Derek Gordon Murray, Frank McSherry, Michael Isard et al. · Communications of the ACM · 2016

We describe the timely dataflow model for distributed computation and its implementation in the Naiad system. The model supports stateful iterative and incremental computations. It enables both low-latency stream processing and high-throughput batch processing, using a new approach to coordination that combines asynchronous and fine-grained synchronous execution. We describe two of the programming frameworks built on Naiad: GraphLINQ for parallel graph processing, and differential dataflow for nested iterative and incremental computations. We show that a general-purpose system can achieve performance that matches, and sometimes exceeds, that of specialized systems.

Cited by 36 works. doi:10.1145/2983551 Open-access PDF

Authors

References