Abstract
We apply state-of-the-art tools in modern high-dimensional numerical linear
algebra to approximate efficiently the spectrum of the Hessian of modern
deepnets, with tens of millions of parameters, trained on real data. Our
results corroborate previous findings, based on small-scale networks, that the
Hessian exhibits "spiked" behavior, with several outliers isolated from a
continuous bulk. We decompose the Hessian into different components and study
the dynamics with training and sample size of each term individually.
Users
Please
log in to take part in the discussion (add own reviews or comments).