MillWheel
Tyler Akidau, Alex Balikov, Kaya Bekiroğlu et al. · Proceedings of the VLDB Endowment · 2013
MillWheel is a framework for building low-latency data-processing applications that is widely used at Google. Users specify a directed computation graph and application code for individual nodes, and the system manages persistent state and the continuous flow of records, all within the envelope of the framework's fault-tolerance guarantees. This paper describes MillWheel's programming model as well as its implementation. The case study of a continuous anomaly detector in use at Google serves to motivate how many of MillWheel's features are used. MillWheel's programming model provides a notion of logical time, making it simple to write time-based aggregations. MillWheel was designed from the outset with fault tolerance and scalability in mind. In practice, we find that MillWheel's unique combination of scalability, fault tolerance, and a versatile programming model lends itself to a wide variety of problems at Google.
Cited by 536 works. doi:10.14778/2536222.2536229
Authors
- Tyler Akidau
- Alex Balikov
- Kaya Bekiroğlu
- Slava Chernyak
- Josh Haberman
- Reuven Lax
- Sam McVeety
- Paul Nordstrom
- Sam Whittle
References
- MapReduce
- Time, clocks, and the ordering of events in a distributed system
- Bigtable
- Virtual time
- Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
- Models and issues in data stream systems
- Aurora: a new model and architecture for data stream management
- The design of the borealis stream processing engine
- Fjording the stream: an architecture for queries over streaming sensor data
- S4: Distributed Stream Computing Platform