We study a deep linear network endowed with a structure. It takes the form of
a matrix $X$ obtained by multiplying $K$ matrices (called factors and
corresponding to the action of the layers). The action of each layer (i.e. a
factor) is obtained by applying a fixed linear operator to a vector of
parameters satisfying a constraint. The number of layers is not limited.
Assuming that $X$ is given and factors have been estimated, the error between
the product of the estimated factors and $X$ (i.e. the reconstruction error) is
either the statistical or the empirical risk. In this paper, we provide
necessary and sufficient conditions on the network topology under which a
stability property holds. The stability property requires that the error on the
parameters defining the factors (i.e. the stability of the recovered
parameters) scales linearly with the reconstruction error (i.e. the risk).
Therefore, under these conditions on the network topology, any successful
learning task leads to stably defined features and therefore interpretable
layers/network.In order to do so, we first evaluate how the Segre embedding and
its inverse distort distances. Then, we show that any deep structured linear
network can be cast as a generic multilinear problem (that uses the Segre
embedding). This is the tensorial lifting. Using the tensorial lifting,
we provide necessary and sufficient conditions for the identifiability of the
factors (up to a scale rearrangement). We finally provide the necessary and
sufficient condition called \NSPlong~(because of the analogy with the usual
Null Space Property in the compressed sensing framework) which guarantees that
the stability property holds. We illustrate the theory with a practical example
where the deep structured linear network is a convolutional linear network. As
expected, the conditions are rather strong but not empty. A simple test on the
network topology can be implemented to test if the condition holds.
Description
[1703.08044] Multilinear compressive sensing and an application to convolutional linear networks
%0 Journal Article
%1 malgouyres2017multilinear
%A Malgouyres, Francois
%A Landsberg, Joseph
%D 2017
%K compression matrix-factorization readings sparsity
%T Multilinear compressive sensing and an application to convolutional
linear networks
%U http://arxiv.org/abs/1703.08044
%X We study a deep linear network endowed with a structure. It takes the form of
a matrix $X$ obtained by multiplying $K$ matrices (called factors and
corresponding to the action of the layers). The action of each layer (i.e. a
factor) is obtained by applying a fixed linear operator to a vector of
parameters satisfying a constraint. The number of layers is not limited.
Assuming that $X$ is given and factors have been estimated, the error between
the product of the estimated factors and $X$ (i.e. the reconstruction error) is
either the statistical or the empirical risk. In this paper, we provide
necessary and sufficient conditions on the network topology under which a
stability property holds. The stability property requires that the error on the
parameters defining the factors (i.e. the stability of the recovered
parameters) scales linearly with the reconstruction error (i.e. the risk).
Therefore, under these conditions on the network topology, any successful
learning task leads to stably defined features and therefore interpretable
layers/network.In order to do so, we first evaluate how the Segre embedding and
its inverse distort distances. Then, we show that any deep structured linear
network can be cast as a generic multilinear problem (that uses the Segre
embedding). This is the tensorial lifting. Using the tensorial lifting,
we provide necessary and sufficient conditions for the identifiability of the
factors (up to a scale rearrangement). We finally provide the necessary and
sufficient condition called \NSPlong~(because of the analogy with the usual
Null Space Property in the compressed sensing framework) which guarantees that
the stability property holds. We illustrate the theory with a practical example
where the deep structured linear network is a convolutional linear network. As
expected, the conditions are rather strong but not empty. A simple test on the
network topology can be implemented to test if the condition holds.
@article{malgouyres2017multilinear,
abstract = {We study a deep linear network endowed with a structure. It takes the form of
a matrix $X$ obtained by multiplying $K$ matrices (called factors and
corresponding to the action of the layers). The action of each layer (i.e. a
factor) is obtained by applying a fixed linear operator to a vector of
parameters satisfying a constraint. The number of layers is not limited.
Assuming that $X$ is given and factors have been estimated, the error between
the product of the estimated factors and $X$ (i.e. the reconstruction error) is
either the statistical or the empirical risk. In this paper, we provide
necessary and sufficient conditions on the network topology under which a
stability property holds. The stability property requires that the error on the
parameters defining the factors (i.e. the stability of the recovered
parameters) scales linearly with the reconstruction error (i.e. the risk).
Therefore, under these conditions on the network topology, any successful
learning task leads to stably defined features and therefore interpretable
layers/network.In order to do so, we first evaluate how the Segre embedding and
its inverse distort distances. Then, we show that any deep structured linear
network can be cast as a generic multilinear problem (that uses the Segre
embedding). This is the {\em tensorial lifting}. Using the tensorial lifting,
we provide necessary and sufficient conditions for the identifiability of the
factors (up to a scale rearrangement). We finally provide the necessary and
sufficient condition called \NSPlong~(because of the analogy with the usual
Null Space Property in the compressed sensing framework) which guarantees that
the stability property holds. We illustrate the theory with a practical example
where the deep structured linear network is a convolutional linear network. As
expected, the conditions are rather strong but not empty. A simple test on the
network topology can be implemented to test if the condition holds.},
added-at = {2019-11-01T15:30:00.000+0100},
author = {Malgouyres, Francois and Landsberg, Joseph},
biburl = {https://www.bibsonomy.org/bibtex/2425ab4d3b4233ed0552f7e7d03f569eb/kirk86},
description = {[1703.08044] Multilinear compressive sensing and an application to convolutional linear networks},
interhash = {8a5780c5432db7a35b9f9e4092474ace},
intrahash = {425ab4d3b4233ed0552f7e7d03f569eb},
keywords = {compression matrix-factorization readings sparsity},
note = {cite arxiv:1703.08044},
timestamp = {2019-11-01T15:30:00.000+0100},
title = {Multilinear compressive sensing and an application to convolutional
linear networks},
url = {http://arxiv.org/abs/1703.08044},
year = 2017
}