Quincy
Michael Isard, Vijayan Prabhakaran, Jon Currey et al. · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · 2009
This paper addresses the problem of scheduling concurrent jobs on clusters where application data is stored on the computing nodes. This setting, in which scheduling computations close to their data is crucial for performance, is increasingly common and arises in systems such as MapReduce, Hadoop, and Dryad as well as many grid-computing environments. We argue that data-intensive computation benefits from a fine-grain resource sharing model that differs from the coarser semi-static resource allocations implemented by most existing cluster computing architectures. The problem of scheduling with locality and fairness constraints has not previously been extensively studied under this resource-sharing model.
Cited by 856 works. doi:10.1145/1629575.1629601
Authors
References
- MapReduce
- Flows in networks
- The Google file system
- Dryad
- The Google file system
- Approximation Algorithms for NP-Hard Problems
- Distributed computing in practice: the Condor experience
- Multiprocessor Scheduling with the Aid of Network Flow Algorithms
- Improving MapReduce performance in heterogeneous environments
- Explicit control a batch-aware distributed file system