@kirk86

A Fine-Grained Spectral Perspective on Neural Networks

, and . (2019)cite arxiv:1907.10599Comment: 12 pages of main text, 14 figures, 39 pages including appendix.

Abstract

Are neural networks biased toward simple functions? Does depth always help learn more complex features? Is training the last layer of a network as good as training all layers? These questions seem unrelated at face value, but in this work we give all of them a common treatment from the spectral perspective. We will study the spectra of the *Conjugate Kernel*, CK, (also called the *Neural Network-Gaussian Process Kernel*), and the *Neural Tangent Kernel*, NTK. Roughly, the CK and the NTK tell us respectively "what a network looks like at initializationänd "what a network looks like during and after training." Their spectra then encode valuable…(more)

Description

[1907.10599] A Fine-Grained Spectral Perspective on Neural Networks

Links and resources

Tags