Curriculum learning
Yoshua Bengio, Jérôme Louradour, Ronan Collobert et al. · International Conference on Machine Learning · 2009
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Cited by 5,002 works. doi:10.1145/1553374.1553380
Authors
References
- A Fast Learning Algorithm for Deep Belief Nets
- Reducing the Dimensionality of Data with Neural Networks
- Greedy Layer-Wise Training of Deep Networks
- Learning Deep Architectures for AI
- Extracting and composing robust features with denoising autoencoders
- Unsupervised learning of distributions on binary vectors using two layer networks
- A unified architecture for natural language processing
- Learning Deep Architectures for AI
- Restricted Boltzmann machines for collaborative filtering
- Efficient Learning of Sparse Representations with an Energy-Based Model