A Practical Bayesian Framework for Backpropagation Networks
David Mackay · Neural Computation · 1992
A quantitative and practical Bayesian framework is described for learning of mappings in feedforward networks. The framework makes possible (1) objective comparisons between solutions using alternative network architectures, (2) objective stopping rules for network pruning or growing procedures, (3) objective choice of magnitude and type of weight decay terms or additive regularizers (for penalizing large weights, etc.), (4) a measure of the effective number of well-determined parameters in a model, (5) quantified estimates of the error bars on network parameters and on network output, and (6) objective comparisons with alternative learning and interpolation models such as splines and radial basis functions. The Bayesian "evidence" automatically embodies "Occam's razor," penalizing overflexible and overcomplex models. The Bayesian approach helps detect poor underlying assumptions in learning models. For learning models well matched to a problem, a good correlation between generalization ability and the Bayesian evidence is obtained.
Cited by 2,911 works. doi:10.1162/neco.1992.4.3.448
Authors
References
- Learning representations by back-propagating errors
- Optimal Brain Damage
- Learning and Relearning in Boltzmann Machines
- Consistent inference of probabilities in layered networks: predictions and generalizations
- Bayesian Interpolation
- The Evidence Framework Applied to Classification Networks
- On the Asymptotic Behaviour of Posterior Distributions
- Exact Calculation of the Hessian Matrix for the Multilayer Perceptron
- The Vapnik-Chervonenkis Dimension: Information versus Complexity in Learning
- Developments in Maximum Entropy Data Analysis