Misc,
Averaging Weights Leads to Wider Optima and Better Generalization
(Aug 8, 2018)
Abstract
Deep neural networks are typically trained by optimizing a loss function with
an SGD variant, in conjunction with a decaying learning rate, until
convergence. We show that simple averaging of multiple points along the
trajectory of SGD, with a cyclical or constant learning rate, leads to better
generalization than conventional training. We also show that this Stochastic
Weight Averaging (SWA) procedure finds much broader optima than SGD, and
approximates the recent Fast Geometric Ensembling (FGE) approach with a single
model. Using SWA we achieve notable improvement in test accuracy over
conventional SGD training on a range of state-of-the-art residual networks,
PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and
ImageNet. In short, SWA is extremely easy to implement, improves
generalization, and has almost no computational overhead.
