Deep AutoRegressive Networks
Karol Gregor, Ivo Danihelka, Andriy Mnih et al. · arXiv (Cornell University) · 2013
We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling. We derive an efficient approximate parameter estimation method based on the minimum description length (MDL) principle, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference. We demonstrate state-of-the-art generative performance on a number of classic data sets: several UCI data sets, MNIST and Atari 2600 games.
Cited by 99 works. doi:10.48550/arxiv.1310.8499 Open-access PDF
Authors
References
- Gradient-based learning applied to document recognition
- UCI Machine Learning Repository
- Extracting and composing robust features with denoising autoencoders
- On the quantitative analysis of deep belief networks
- Modeling by shortest data description
- Generating Sequences With Recurrent Neural Networks
- Simple statistical gradient-following algorithms for connectionist reinforcement learning
- Information theory, inference, and learning algorithms
- Deep Boltzmann machines
- Sparse Feature Learning for Deep Belief Networks