Flat Minima
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997
We present a new algorithm for finding low-complexity neural networks with high generalization capability. The algorithm searches for a "flat" minimum of the error function. A flat minimum is a large connected region in weight space where the error remains approximately constant. An MDL-based, Bayesian argument suggests that flat minima correspond to "simple" networks and low expected overfitting. The argument is based on a Gibbs algorithm variant and a novel way of splitting generalization error into underfitting and overfitting error. Unlike many previous approaches, ours does not require gaussian assumptions and does not depend on a "good" weight prior. Instead we have a prior over input-output functions, thus taking into account net architecture and training set. Although our algorithm requires the computation of second-order derivatives, it has backpropagation's order of complexity. Automatically, it effectively prunes units, weights, and input lines. Various experiments with feedforward and recurrent nets are described. In an application to stock market prediction, flat minimum search outperforms conventional backprop, weight decay, and "optimal brain surgeon/optimal brain damage".
Cited by 676 works. doi:10.1162/neco.1997.9.1.1
Authors
References
- A Mathematical Theory of Communication
- Cross-Validatory Choice and Assessment of Statistical Predictions
- Advances in Neural Information Processing Systems 37
- Modeling by shortest data description
- A Practical Bayesian Framework for Backpropagation Networks
- Smoothing noisy data with spline functions
- Bayesian Interpolation
- Statistical Predictor Identification
- Statistical predictor identification
- Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter