Learning to reinforcement learn
Jane X. Wang, Zeb L. Kurth-Nelson, Dhruva Tirumala et al. · arXiv (Cornell University) · 2016
In recent years deep reinforcement learning (RL) systems have attained superhuman performance in a number of challenging task domains. However, a major limitation of such applications is their demand for massive amounts of training data. A critical present objective is thus to develop deep RL methods that can adapt rapidly to new tasks. In the present work we introduce a novel approach to this challenge, which we refer to as deep meta-reinforcement learning. Previous work has shown that recurrent networks can support meta-learning in a fully supervised context. We extend this approach to the RL setting. What emerges is a system that is trained using one RL algorithm, but whose recurrent dynamics implement a second, quite separate RL procedure. This second, learned RL algorithm can differ from the original one in arbitrary ways. Importantly, because it is learned, it is configured to exploit structure in the training domain. We unpack these points in a series of seven proof-of-concept experiments, each of which examines a key aspect of deep meta-RL. We consider prospects for extending and scaling up the approach, and also point out some potentially important implications for neuroscience.
Cited by 325 works. doi:10.48550/arxiv.1611.05763 Open-access PDF
Authors
- Jane X. Wang
- Zeb L. Kurth-Nelson
- Dhruva Tirumala
- Hubert Soyer
- Joel Z. Leibo
- Rémi Munos
- Charles Blundell
- Dharshan Kumaran
- Matthew Botvinick
References
- ImageNet: A large-scale hierarchical image database
- Deep learning
- Human-level control through deep reinforcement learning
- Mastering the game of Go with deep neural networks and tree search
- The formation of learning sets.
- Neural Architecture Search with Reinforcement Learning
- ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES
- Bandit Processes and Dynamic Allocation Indices
- Building machines that learn and think like people
- Learning to Learn Using Gradient Descent