Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
Sergey Ioffe · arXiv (Cornell University) · 2017
Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches. At the same time, Batch Renormalization retains the benefits of batchnorm such as insensitivity to initialization and training efficiency.
Cited by 367 works. doi:10.48550/arxiv.1702.03275 Open-access PDF
Authors
References
- Deep Residual Learning for Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Rethinking the Inception Architecture for Computer Vision
- Improved Techniques for Training GANs
- Neighbourhood Components Analysis
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Revisiting Distributed Synchronous SGD
- Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks