Применение машинного обучения для прогнозирования времени выполнения суперкомпьютерных заданий
Anton V. Baranov, Баранов А.В., Д.С. Николаев et al. · Международный журнал Программные продукты и системы · 2020
The authors devoted the paper to machine learning methods and algorithms for the supercomputer job execution time predicting.The supercomputer systems statistics for multiple-access shows that the actual execution time for most jobs substantially diverges from the time requested by the user.This reduces the scheduling job efficiency because inaccurate estimates of execution time lead to a suboptimal schedule for job launches.The paper considers the job classification, which is based on actual time for job performance to the requested one.There were six job classes moreover the ratio of the actual job completion time to the requested time for each class differs by an order of magnitude from the previous class.Statistical data from a shared-use supercomputer system is the basis for predicting the job completion time by assigning the incoming job to one of the classes.The supercomputer RIKEN Integrated Cluster of Clusters (RICC) statistics presented in SWF format were as initial data.The analyzing results for these statistics allowed us to identify significant features of the job flow for machine learning.There was a feature ranking by importance and there were hidden patterns that affect the forecasting accuracy, in particular, there was a mutual correlation for selected features.For common machine learning algorithms, such as logistic regression, decision tree, k nearest neighbors, linear discriminant analysis, support vector method, random forest, gradient boosting, and direct propagation neural network, there were estimates of the probability of a correct forecast.Algorithms of the decision tree, random forest, and direct propagation neural networks showed the best values.
Cited by 4 works. doi:10.15827/0236-235x.130.218-228 Open-access PDF
Authors
References
- Greedy function approximation: A gradient boosting machine.
- An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression
- Discriminant Analysis and Statistical Pattern Recognition
- SLURM: Simple Linux Utility for Resource Management
- Job scheduling under the Portable Batch System
- Logistic Regression
- Benchmarks and Standards for the Evaluation of Parallel Job Schedulers
- Scalable system scheduling for HPC and big data
- Diagnosing Performance Variations in HPC Applications Using Machine Learning
- Introduction to Scikit-Learn