Online Learning for Matrix Factorization and Sparse Coding
Julien Mairal, Francis Bach, Jean Ponce et al. · arXiv (Cornell University) · 2009
Sparse coding--that is, modelling data vectors as sparse linear combinations of basis elements--is widely used in machine learning, neuroscience, signal processing, and statistics. This paper focuses on the large-scale matrix factorization problem that consists of learning the basis set, adapting it to specific data. Variations of this problem include dictionary learning in signal processing, non-negative matrix factorization and sparse principal component analysis. In this paper, we propose to address these tasks with a new online optimization algorithm, based on stochastic approximations, which scales up gracefully to large datasets with millions of training samples, and extends naturally to various matrix factorization formulations, making it suitable for a wide range of learning problems. A proof of convergence is presented, along with experiments with natural images and genomic data demonstrating that it leads to state-of-the-art performance in terms of speed and optimization for both small and large datasets.
Cited by 2,258 works. doi:10.48550/arxiv.0908.0050 Open-access PDF
Authors
References
- Matrix Computations
- Regression Shrinkage and Selection Via the Lasso
- A Wavelet Tour of Signal Processing
- Nonlinear Programming
- Atomic Decomposition by Basis Pursuit
- Least angle regression
- RELATIONS BETWEEN TWO SETS OF VARIATES
- Regularization and Variable Selection Via the Elastic Net
- A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
- Sparse coding with an overcomplete basis set: A strategy employed by V1?