Efficiently Building a Large Scale Dataset for Program Induction
Lauren Milechin, Javier Lopez-Contreras, Ferran Alet · IEEE High Performance Extreme Computing Conference · 2021
One of the applications of machine learning with the most potential is speeding up expensive computational processes, i.e. learning to think. To do so, one first generates a large-scale dataset by a compute-intensive process and then trains a model to approximate the distribution. High performance computing (HPC) is a perfect fit for these processes, as one may efficiently deploy large amounts of computation to generate a dataset in a reasonable amount of time, to then learn a computationally-efficient solution. Here, we focus on generating a program synthesis dataset. Finding the program that fits a given input-output specification is very expensive, but generating the input-output pairs for a given program is a well-defined process. In this work, we show how we efficiently ran hundreds of thousands of C++ codes line-by-line and used intermediate variable states to generate a large-scale program synthesis dataset.
Cited by 1 works. doi:10.1109/hpec49654.2021.9622815
Authors
References
- ImageNet: A large-scale hierarchical image database
- Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics
- Mastering the game of Go without human knowledge
- SLURM: Simple Linux Utility for Resource Management
- The Lustre Storage Architecture
- Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis
- Graph-based, Self-Supervised Program Repair from Diagnostic Feedback
- LLMapReduce: Multi-level map-reduce for high performance data analysis
- Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs
- A large-scale benchmark for few-shot program induction and synthesis