Explicit control a batch-aware distributed file system
John Bent, Douglas Thain, Andrea Carol Arpaci-Dusseau et al. · 2004
We present the design, implementation, and evaluation of the Batch-Aware Distributed File System (BAD-FS), a system designed to orchestrate large, I/O-intensive batch workloads on remote computing clusters distributed across the wide area. BAD-FS consists of two novel components: a storage layer that exposes control of traditionally fixed policies such as caching, consistency, and replication; and a scheduler that exploits this control as necessary for different workloads. By extracting control from the storage layer and placing it within an external scheduler, BAD-FS manages both storage and computation in a coordinated way while gracefully dealing with cache consistency, fault-tolerance, and space management issues in a workload-specific manner. Using both microbenchmarks and real workloads, we demonstrate the performance benefits of explicit control, delivering excellent end-to-end performance across the wide-area.
Cited by 114 works.
Authors
References
- Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
- A relational model of data for large shared data banks
- The Google file system
- The Anatomy of the Grid: Enabling Scalable Virtual Organizations
- Scale and performance in a distributed file system
- Database mining: a performance perspective
- OceanStore
- Condor-a hunter of idle workstations
- Resilient overlay networks
- Wide-area cooperative storage with CFS