Abstract
Over-parametrization of deep neural networks has recently been shown to be
key to their successful training. However, it also renders them prone to
overfitting and makes them expensive to store and train. Tensor regression
networks significantly reduce the number of effective parameters in deep neural
networks while retaining accuracy and the ease of training. They replace the
flattening and fully-connected layers with a tensor regression layer, where the
regression weights are expressed through the factors of a low-rank tensor
decomposition. In this paper, to further improve tensor regression networks, we
propose a novel stochastic rank-regularization. It consists of a novel
randomized tensor sketching method to approximate the weights of tensor
regression layers. We theoretically and empirically establish the link between
our proposed stochastic rank-regularization and the dropout on low-rank tensor
regression. Extensive experimental results with both synthetic data and real
world datasets (i.e., CIFAR-100 and the UK Biobank brain MRI dataset) support
that the proposed approach i) improves performance in both classification and
regression tasks, ii) decreases overfitting, iii) leads to more stable training
and iv) improves robustness to adversarial attacks and random noise.
Users
Please
log in to take part in the discussion (add own reviews or comments).