Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
Takeru Miyato, Shin‐ichi Maeda, Masanori Koyama et al. · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2018
We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input. Virtual adversarial loss is defined as the robustness of the conditional label distribution around each input data point against local perturbation. Unlike adversarial training, our method defines the adversarial direction without label information and is hence applicable to semi-supervised learning. Because the directions in which we smooth the model are only "virtually" adversarial, we call our method virtual adversarial training (VAT). The computational cost of VAT is relatively low. For neural networks, the approximated gradient of virtual adversarial loss can be computed with no more than two pairs of forward- and back-propagations. In our experiments, we applied VAT to supervised and semi-supervised learning tasks on multiple benchmark datasets. With a simple enhancement of the algorithm based on the entropy minimization principle, our VAT achieves state-of-the-art performance for semi-supervised learning tasks on SVHN and CIFAR-10.
Cited by 2,843 works. doi:10.1109/tpami.2018.2858821 Open-access PDF
Authors
References
- Adam: A Method for Stochastic Optimization
- AutoResearch Lite: a reproducible paper-inspired experiment runner (Adam mechanism reproduction and convergence-speed comparison)
- Learning Multiple Layers of Features from Tiny Images
- Dropout: a simple way to prevent neural networks from overfitting
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Densely Connected Convolutional Networks
- Rectified Linear Units Improve Restricted Boltzmann Machines
- Information Theory and an Extension of the Maximum Likelihood Principle
- Book Review: Pattern Recognition and Machine Learning
- GAN(Generative Adversarial Nets)