Universal Value Function Approximators
Tom Schaul, Dan Horgan, Karol Gregor et al. · 2015
Value functions are a core component of reinforcement learning systems. The main idea is to to construct a single function approximator V (s; θ) that estimates the long-term reward from any state s, using parameters θ. In this paper we introduce universal value function approximators (UVFAs) V (s, g; θ) that generalise not just over states s but also over goals g. We develop an efficient technique for supervised learning of UVFAs, by factoring observed values into separate embedding vectors for state and goal, and then learning a mapping from s and g to these factored embedding vectors. We show how this technique may be incorporated into a reinforcement learning algorithm that updates the UVFA solely from observed rewards. Finally, we demonstrate that a UVFA can successfully generalise to previously unseen goals.
Cited by 541 works.
Authors
References
- AutoResearch Lite: a reproducible paper-inspired experiment runner (Adam mechanism reproduction and convergence-speed comparison)
- Visualizing Data using t-SNE
- Distributed Representations of Words and Phrases and their Compositionality
- Torch7: A Matlab-like Environment for Machine Learning
- Introduction to Reinforcement Learning
- Natural Language Processing (almost) from Scratch
- Multitask Learning
- Playing Atari with Deep Reinforcement Learning
- Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
- Hierarchical Learning in Stochastic Domains: Preliminary Results