Data-driven batch scheduling
John Bent, Timothy E. Denehy, Miron Livny et al. · 2009
In this paper, we develop data-driven strategies for batch computing schedulers. Current CPU-centric batch schedulers ignore the data needs within workloads and execute them by linking them transparently and directly to their needed data. When scheduled on remote computational resources, this elegant solution of direct data access can incur an order of magnitude performance penalty for data-intensive workloads. Adding data-awareness to batch schedulers allows a careful coordination of data and CPU allocation thereby reducing the cost of remote execution. We offer here new techniques by which batch schedulers can become data-driven. Such systems can use our analytical predictive models to select one of the four data-driven scheduling policies that we have created. Through simulation, we demonstrate the accuracy of our predictive models and show how they can reduce time to completion for some workloads by as much as 80%.
Cited by 18 works. doi:10.1145/1552280.1552281
Authors
References
- Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
- The Google file system
- The Anatomy of the Grid: Enabling Scalable Virtual Organizations
- Database mining: a performance perspective
- OceanStore
- Condor-a hunter of idle workstations
- The working set model for program behavior
- Wide-area cooperative storage with CFS
- A trace-driven analysis of the UNIX 4.2 BSD file system
- The working set model for program behavior