MapReduce
Jeffrey Dean, Sanjay Ghemawat · Communications of the ACM · 2008
MapReduce is a programming model and an associated implementation for processing and generating large datasets that is amenable to a broad variety of real-world tasks. Users specify the computation in terms of a map and a reduce function, and the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks. Programmers find the system easy to use: more than ten thousand distinct MapReduce programs have been implemented internally at Google over the past four years, and an average of one hundred thousand MapReduce jobs are executed on Google's clusters every day, processing a total of more than twenty petabytes of data per day.
Cited by 18,457 works. doi:10.1145/1327452.1327492 Open-access PDF
Authors
References
- Lecture Notes in Computer Science 1205
- The Google file system
- MapReduce: Simplified Data Processing on Large Cluster
- Using MPI: Portable parallel programming with the message-passing interface
- The Google file system
- A Bridging Model for Parallel Computation
- Web search for a planet: the google cluster architecture
- Efficient dispersal of information for security, load balancing, and fault tolerance
- Parallel Prefix Computation
- Map-Reduce for Machine Learning on Multicore