Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen et al. · arXiv (Cornell University) · 2020
We present a new method for evaluating and training unnormalized density models. Our approach only requires access to the gradient of the unnormalized model's log-density. We estimate the Stein discrepancy between the data density $p(x)$ and the model density $q(x)$ defined by a vector function of the data. We parameterize this function with a neural network and fit its parameters to maximize the discrepancy. This yields a novel goodness-of-fit test which outperforms existing methods on high dimensional data. Furthermore, optimizing $q(x)$ to minimize this discrepancy produces a novel method for training unnormalized models which scales more gracefully than existing methods. The ability to both learn and compare models is a unique feature of the proposed method.
Cited by 29 works. doi:10.48550/arxiv.2002.05616 Open-access PDF
Authors
References
- Adam: A Method for Stochastic Optimization
- A Fast Learning Algorithm for Deep Belief Nets
- Training Products of Experts by Minimizing Contrastive Divergence
- GAN(Generative Adversarial Nets)
- Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- A kernel two-sample test
- A Bound for the Error in the Normal Approximation to the Distribution of a Sum of Dependent Random Variables
- A note on the evaluation of generative models