I/O acceleration with pattern detection
Jun He, John Bent, Aaron Torres et al. · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · 2013
The I/O bottleneck in high-performance computing is becoming worse as application data continues to grow. In this work, we explore how patterns of I/O within these applications can significantly affect the effectiveness of the underlying storage systems and how these same patterns can be utilized to improve many aspects of the I/O stack and mitigate the I/O bottleneck. We offer three main contributions in this paper. First, we develop and evaluate algorithms by which I/O patterns can be efficiently discovered and described. Second, we implement one such algorithm to reduce the metadata quantity in a virtual parallel file system by up to several orders of magnitude, thereby increasing the performance of writes and reads by up to 40 and 480 percent respectively. Third, we build a prototype file system with pattern-aware prefetching and evaluate it to show a 46 percent reduction in I/O latency. Finally, we believe that efficient pattern discovery and description, coupled with the observed predictability of complex patterns within many high-performance applications, offers significant potential to enable many additional I/O optimizations.
Cited by 67 works. doi:10.1145/2462902.2462909
Authors
References
- A universal algorithm for sequential data compression
- Fast Pattern Matching in Strings
- Informed prefetching and caching
- On implementing MPI-IO portably and with high performance
- Information and control in gray-box systems
- File-access characteristics of parallel scientific workloads
- Automatic I/O Hint Generation through Speculative Execution
- PLFS
- Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS)
- A static analysis of I/O characteristics of scientific applications in a production workload