Predicting Parameters in Deep Learning
Misha Denil, Babak Shakibi, Laurent Dinh et al. · arXiv (Cornell University) · 2013
We demonstrate that there is significant redundancy in the parameterization of several deep learning models. Given only a few weight values for each feature it is possible to accurately predict the remaining values. Moreover, we show that not only can the parameter values be predicted, but many of them need not be learned at all. We train several different architectures by learning only a small number of weights and predicting the rest. In the best case we are able to predict more than 95% of the weights of a network without any drop in accuracy.
Cited by 663 works. doi:10.48550/arxiv.1306.0543 Open-access PDF
Authors
References
- Gradient-based learning applied to document recognition
- Learning representations by back-propagating errors
- Improving neural networks by preventing co-adaptation of feature detectors
- Large Scale Distributed Deep Networks
- Optimal Brain Damage
- Kernel Methods for Pattern Analysis
- High-Performance Neural Networks for Visual Object Classification
- An analysis of single-layer networks in unsupervised feature learning
- Building high-level features using large scale unsupervised learning
- Speaker-independent phone recognition using hidden Markov models