Tensorizing Neural Networks
Alexander Novikov, Dmitry Podoprikhin, Anton Osokin et al. · arXiv (Cornell University) · 2015
Deep neural networks currently demonstrate state-of-the-art performance in several domains. At the same time, models of this class are very demanding in terms of computational resources. In particular, a large amount of memory is required by commonly used fully-connected layers, making it hard to use the models on low-end devices and stopping the further increase of the model size. In this paper we convert the dense weight matrices of the fully-connected layers to the Tensor Train format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved. In particular, for the Very Deep VGG networks we report the compression factor of the dense weight matrix of a fully-connected layer up to 200000 times leading to the compression factor of the whole network up to 7 times.
Cited by 558 works. doi:10.48550/arxiv.1509.06569 Open-access PDF
Authors
References
- ImageNet classification with deep convolutional neural networks
- Learning representations by back-propagating errors
- ImageNet Large Scale Visual Recognition Challenge
- Learning Multiple Layers of Features from Tiny Images
- Approximation by superpositions of a sigmoidal function
- Practical Bayesian Optimization of Machine Learning Algorithms
- Analysis of Individual Differences in Multidimensional Scaling Via an N-way Generalization of “Eckart-Young” Decomposition
- Some Mathematical Notes on Three-Mode Factor Analysis
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- MatConvNet