Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
Will Grathwohl, Dami Choi, Yuhuai Wu et al. · arXiv (Cornell University) · 2017
Gradient-based optimization is the foundation of deep learning and reinforcement learning. Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy. We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables. Our method uses gradients of a neural network trained jointly with model parameters or policies, and is applicable in both discrete and continuous settings. We demonstrate this framework for training discrete latent-variable models. We also give an unbiased, action-conditional extension of the advantage actor-critic reinforcement learning algorithm.
Cited by 90 works. doi:10.48550/arxiv.1711.00123 Open-access PDF
Authors
References
- AutoResearch Lite: a reproducible paper-inspired experiment runner (Adam mechanism reproduction and convergence-speed comparison)
- Auto-Encoding Variational Bayes
- Simple statistical gradient-following algorithms for connectionist reinforcement learning
- Policy Gradient Methods for Reinforcement Learning with Function Approximation
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Compiling fast partial derivatives of functions given by algorithms
- Human-level concept learning through probabilistic program induction
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Categorical Reparameterization with Gumbel-Softmax
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables