A multiplicative constant scaling factor is often applied to the model output
to adjust the dynamics of neural network parameters. This has been used as one
of the key interventions in an empirical study of lazy and active behavior.
However, we show that the combination of such scaling and a commonly used
adaptive learning rate optimizer strongly affects the training behavior of the
neural network. This is problematic as it can cause unintended behavior
of neural networks, resulting in the misinterpretation of experimental results.
Specifically, for some scaling settings, the effect of the adaptive learning
rate disappears or is strongly influenced by the scaling factor. To avoid the
unintended effect, we present a modification of an optimization algorithm and
demonstrate remarkable differences between adaptive learning rate optimization
and simple gradient descent, especially with a small ($<1.0$) scaling factor.
Description
Unintended Effects on Adaptive Learning Rate for Training Neural Network with Output Scale Change
%0 Generic
%1 kanoh2021unintended
%A Kanoh, Ryuichi
%A Sugiyama, Mahito
%D 2021
%K Effects adaptive learning-rate
%T Unintended Effects on Adaptive Learning Rate for Training Neural Network
with Output Scale Change
%U http://arxiv.org/abs/2103.03466
%X A multiplicative constant scaling factor is often applied to the model output
to adjust the dynamics of neural network parameters. This has been used as one
of the key interventions in an empirical study of lazy and active behavior.
However, we show that the combination of such scaling and a commonly used
adaptive learning rate optimizer strongly affects the training behavior of the
neural network. This is problematic as it can cause unintended behavior
of neural networks, resulting in the misinterpretation of experimental results.
Specifically, for some scaling settings, the effect of the adaptive learning
rate disappears or is strongly influenced by the scaling factor. To avoid the
unintended effect, we present a modification of an optimization algorithm and
demonstrate remarkable differences between adaptive learning rate optimization
and simple gradient descent, especially with a small ($<1.0$) scaling factor.
@misc{kanoh2021unintended,
abstract = {A multiplicative constant scaling factor is often applied to the model output
to adjust the dynamics of neural network parameters. This has been used as one
of the key interventions in an empirical study of lazy and active behavior.
However, we show that the combination of such scaling and a commonly used
adaptive learning rate optimizer strongly affects the training behavior of the
neural network. This is problematic as it can cause \emph{unintended behavior}
of neural networks, resulting in the misinterpretation of experimental results.
Specifically, for some scaling settings, the effect of the adaptive learning
rate disappears or is strongly influenced by the scaling factor. To avoid the
unintended effect, we present a modification of an optimization algorithm and
demonstrate remarkable differences between adaptive learning rate optimization
and simple gradient descent, especially with a small ($<1.0$) scaling factor.},
added-at = {2021-07-06T21:52:42.000+0200},
author = {Kanoh, Ryuichi and Sugiyama, Mahito},
biburl = {https://www.bibsonomy.org/bibtex/2cb4049bf38e2c2b5ab3c8c033f65af95/rcb},
description = {Unintended Effects on Adaptive Learning Rate for Training Neural Network with Output Scale Change},
interhash = {0985151286834d0fd818c6538ad53e2f},
intrahash = {cb4049bf38e2c2b5ab3c8c033f65af95},
keywords = {Effects adaptive learning-rate},
note = {cite arxiv:2103.03466},
timestamp = {2021-07-06T21:52:42.000+0200},
title = {Unintended Effects on Adaptive Learning Rate for Training Neural Network
with Output Scale Change},
url = {http://arxiv.org/abs/2103.03466},
year = 2021
}