Meta-learning curiosity algorithms
Ferran Alet, Schneider, Martin F., Tomás Lozano‐Pérez et al. · arXiv (Cornell University) · 2020
We hypothesize that curiosity is a mechanism found by evolution that encourages meaningful exploration early in an agent's life in order to expose it to experiences that enable it to obtain high rewards over the course of its lifetime. We formulate the problem of generating curious behavior as one of meta-learning: an outer loop will search over a space of curiosity mechanisms that dynamically adapt the agent's reward signal, and an inner loop will perform standard reinforcement learning using the adapted reward signal. However, current meta-RL methods based on transferring neural network weights have only generalized between very similar tasks. To broaden the generalization, we instead propose to meta-learn algorithms: pieces of code similar to those designed by humans in ML papers. Our rich language of programs combines neural networks with other building blocks such as buffers, nearest-neighbor modules and custom loss functions. We demonstrate the effectiveness of the approach empirically, finding two novel curiosity algorithms that perform on par or better than human-designed published curiosity algorithms in domains as disparate as grid navigation with image inputs, acrobot, lunar lander, ant and hopper.
Cited by 15 works. doi:10.48550/arxiv.2003.05325 Open-access PDF
Authors
References
- AutoResearch Lite: a reproducible paper-inspired experiment runner (Adam mechanism reproduction and convergence-speed comparison)
- Automatic differentiation in PyTorch
- Diagnosing Non-Intermittent Anomalies in Reinforcement Learning Policy Executions (Short Paper)
- Learning Transferable Architectures for Scalable Image Recognition
- Neural Architecture Search with Reinforcement Learning
- MuJoCo: A physics engine for model-based control
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Evolving Neural Networks through Augmenting Topologies
- Efficient Neural Architecture Search via Parameters Sharing
- Curiosity-Driven Exploration by Self-Supervised Prediction