MADE: Masked Autoencoder for Distribution Estimation
M. Germain, K. Gregor, I. Murray, and H. Larochelle. (2015)cite arxiv:1502.03509Comment: 9 pages and 1 page of supplementary material. Updated to match published version.
Abstract
There has been a lot of recent interest in designing neural network models to
estimate a distribution from a set of examples. We introduce a simple
modification for autoencoder neural networks that yields powerful generative
models. Our method masks the autoencoder's parameters to respect autoregressive
constraints: each input is reconstructed only from previous inputs in a given
ordering. Constrained this way, the autoencoder outputs can be interpreted as a
set of conditional probabilities, and their product, the full joint
probability. We can also train a single network that can decompose the joint
probability in multiple different orderings. Our simple framework can be
applied to multiple architectures, including deep ones. Vectorized
implementations, such as on GPUs, are simple and fast. Experiments demonstrate
that this approach is competitive with state-of-the-art tractable distribution
estimators. At test time, the method is significantly faster and scales better
than other autoregressive estimators.
Description
[1502.03509] MADE: Masked Autoencoder for Distribution Estimation
%0 Journal Article
%1 germain2015masked
%A Germain, Mathieu
%A Gregor, Karol
%A Murray, Iain
%A Larochelle, Hugo
%D 2015
%K flows generative-models
%T MADE: Masked Autoencoder for Distribution Estimation
%U http://arxiv.org/abs/1502.03509
%X There has been a lot of recent interest in designing neural network models to
estimate a distribution from a set of examples. We introduce a simple
modification for autoencoder neural networks that yields powerful generative
models. Our method masks the autoencoder's parameters to respect autoregressive
constraints: each input is reconstructed only from previous inputs in a given
ordering. Constrained this way, the autoencoder outputs can be interpreted as a
set of conditional probabilities, and their product, the full joint
probability. We can also train a single network that can decompose the joint
probability in multiple different orderings. Our simple framework can be
applied to multiple architectures, including deep ones. Vectorized
implementations, such as on GPUs, are simple and fast. Experiments demonstrate
that this approach is competitive with state-of-the-art tractable distribution
estimators. At test time, the method is significantly faster and scales better
than other autoregressive estimators.
@article{germain2015masked,
abstract = {There has been a lot of recent interest in designing neural network models to
estimate a distribution from a set of examples. We introduce a simple
modification for autoencoder neural networks that yields powerful generative
models. Our method masks the autoencoder's parameters to respect autoregressive
constraints: each input is reconstructed only from previous inputs in a given
ordering. Constrained this way, the autoencoder outputs can be interpreted as a
set of conditional probabilities, and their product, the full joint
probability. We can also train a single network that can decompose the joint
probability in multiple different orderings. Our simple framework can be
applied to multiple architectures, including deep ones. Vectorized
implementations, such as on GPUs, are simple and fast. Experiments demonstrate
that this approach is competitive with state-of-the-art tractable distribution
estimators. At test time, the method is significantly faster and scales better
than other autoregressive estimators.},
added-at = {2019-12-09T11:32:16.000+0100},
author = {Germain, Mathieu and Gregor, Karol and Murray, Iain and Larochelle, Hugo},
biburl = {https://www.bibsonomy.org/bibtex/212cd18986f952c962f9d40daca33527c/kirk86},
description = {[1502.03509] MADE: Masked Autoencoder for Distribution Estimation},
interhash = {a747418f3a2f93767a03aa0faf7a7e0e},
intrahash = {12cd18986f952c962f9d40daca33527c},
keywords = {flows generative-models},
note = {cite arxiv:1502.03509Comment: 9 pages and 1 page of supplementary material. Updated to match published version},
timestamp = {2019-12-09T11:32:16.000+0100},
title = {MADE: Masked Autoencoder for Distribution Estimation},
url = {http://arxiv.org/abs/1502.03509},
year = 2015
}