MapReduce online
Tyson Condie, Neil Conway, Peter Alvaro et al. · 2010
MapReduce is a popular framework for data-intensive distributed computing of batch jobs. To simplify fault tolerance, many implementations of MapReduce materialize the entire output of each map and reduce task before it can be consumed. In this paper, we propose a modified MapReduce architecture that allows data to be pipelined between operators. This extends the MapReduce programming model beyond batch processing, and can reduce completion times and improve system utilization for batch jobs as well. We present a modified version of the Hadoop MapReduce framework that supports online aggregation, which allows users to see early returns from a job as it is being computed. Our Hadoop Online Prototype (HOP) also supports continuous queries, which enable MapReduce programs to be written for applications such as event monitoring and stream processing. HOP retains the fault tolerance properties of Hadoop and can run unmodified user-defined MapReduce programs.
Cited by 673 works. doi:10.5555/1855711.1855732
Authors
References
- MapReduce
- Dryad
- Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals
- Pig latin
- A comparison of approaches to large-scale data analysis
- Interpreting the Data: Parallel Analysis with Sawzall
- Online aggregation
- Hive
- TelegraphCQ: Continuous Dataflow Processing for an Uncertain World.
- New sampling-based summary statistics for improving approximate query answers