Learning Fast Approximations of Sparse Coding
Karol Gregor, Yann LeCun · 2010
In Sparse Coding (SC), input vectors are reconstructed using a sparse linear combination of basis vectors. SC has become a popular method for extracting features from data. For a given input, SC minimizes a quadratic reconstruction error with an L1 penalty term on the code. The process is often too slow for applications such as real-time pattern recognition. We proposed two versions of a very fast algorithm that produces approximate estimates of the sparse code that can be used to compute good visual features, or to initialize exact iterative algorithms. The main idea is to train a non-linear, feed-forward predictor with a specific architecture and a fixed depth to produce the best possible approximation of the sparse code. A version of the method, which can be seen as a trainable version of Li and Osher’s coordinate descent method, is shown to produce approximate solutions with 10 times less computation than Li and Osher’s for the same approximation error. Unlike previous proposals for sparse code predictors, the system allows a kind of approximate “explaining away ” to take place during inference. The resulting predictor is differentiable and can be included into globallytrained recognition systems. 1.
Cited by 1,132 works.
Authors
References
- What is the best multi-stage architecture for object recognition?
- Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations
- Emergence of simple-cell receptive field properties by learning a sparse code for natural images
- Linear spatial pyramid matching using sparse coding for image classification
- Atomic Decomposition by Basis Pursuit
- Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ 1 minimization
- Efficient sparse coding algorithms
- Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
- An iterative thresholding algorithm for linear inverse problems with a sparsity constraint
- Sparse deep belief net model for visual area V2