Scene Representation Networks: Continuous 3D-Structure-Aware Neural
Scene Representations
V. Sitzmann, M. Zollhöfer, and G. Wetzstein. (2019)cite arxiv:1906.01618Comment: Video: https://youtu.be/6vMEBWD8O20 Project page: https://vsitzmann.github.io/srns/.
Abstract
Unsupervised learning with generative models has the potential of discovering
rich representations of 3D scenes. While geometric deep learning has explored
3D-structure-aware representations of scene geometry, these models typically
require explicit 3D supervision. Emerging neural scene representations can be
trained only with posed 2D images, but existing methods ignore the
three-dimensional structure of scenes. We propose Scene Representation Networks
(SRNs), a continuous, 3D-structure-aware scene representation that encodes both
geometry and appearance. SRNs represent scenes as continuous functions that map
world coordinates to a feature representation of local scene properties. By
formulating the image formation as a differentiable ray-marching algorithm,
SRNs can be trained end-to-end from only 2D images and their camera poses,
without access to depth or shape. This formulation naturally generalizes across
scenes, learning powerful geometry and appearance priors in the process. We
demonstrate the potential of SRNs by evaluating them for novel view synthesis,
few-shot reconstruction, joint shape and appearance interpolation, and
unsupervised discovery of a non-rigid face model.
Description
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
%0 Generic
%1 sitzmann2019scene
%A Sitzmann, Vincent
%A Zollhöfer, Michael
%A Wetzstein, Gordon
%D 2019
%K 3d_reconstruction neural_reconstruction neural_rendering nips2019 scene_representation
%T Scene Representation Networks: Continuous 3D-Structure-Aware Neural
Scene Representations
%U http://arxiv.org/abs/1906.01618
%X Unsupervised learning with generative models has the potential of discovering
rich representations of 3D scenes. While geometric deep learning has explored
3D-structure-aware representations of scene geometry, these models typically
require explicit 3D supervision. Emerging neural scene representations can be
trained only with posed 2D images, but existing methods ignore the
three-dimensional structure of scenes. We propose Scene Representation Networks
(SRNs), a continuous, 3D-structure-aware scene representation that encodes both
geometry and appearance. SRNs represent scenes as continuous functions that map
world coordinates to a feature representation of local scene properties. By
formulating the image formation as a differentiable ray-marching algorithm,
SRNs can be trained end-to-end from only 2D images and their camera poses,
without access to depth or shape. This formulation naturally generalizes across
scenes, learning powerful geometry and appearance priors in the process. We
demonstrate the potential of SRNs by evaluating them for novel view synthesis,
few-shot reconstruction, joint shape and appearance interpolation, and
unsupervised discovery of a non-rigid face model.
@misc{sitzmann2019scene,
abstract = {Unsupervised learning with generative models has the potential of discovering
rich representations of 3D scenes. While geometric deep learning has explored
3D-structure-aware representations of scene geometry, these models typically
require explicit 3D supervision. Emerging neural scene representations can be
trained only with posed 2D images, but existing methods ignore the
three-dimensional structure of scenes. We propose Scene Representation Networks
(SRNs), a continuous, 3D-structure-aware scene representation that encodes both
geometry and appearance. SRNs represent scenes as continuous functions that map
world coordinates to a feature representation of local scene properties. By
formulating the image formation as a differentiable ray-marching algorithm,
SRNs can be trained end-to-end from only 2D images and their camera poses,
without access to depth or shape. This formulation naturally generalizes across
scenes, learning powerful geometry and appearance priors in the process. We
demonstrate the potential of SRNs by evaluating them for novel view synthesis,
few-shot reconstruction, joint shape and appearance interpolation, and
unsupervised discovery of a non-rigid face model.},
added-at = {2020-11-25T13:01:52.000+0100},
author = {Sitzmann, Vincent and Zollhöfer, Michael and Wetzstein, Gordon},
biburl = {https://www.bibsonomy.org/bibtex/258a7f4013c3a6ec459b6cf174bc754b3/shuncheng.wu},
description = {Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations},
interhash = {2fb38ce7a492ef9971d1774cb875a1ce},
intrahash = {58a7f4013c3a6ec459b6cf174bc754b3},
keywords = {3d_reconstruction neural_reconstruction neural_rendering nips2019 scene_representation},
note = {cite arxiv:1906.01618Comment: Video: https://youtu.be/6vMEBWD8O20 Project page: https://vsitzmann.github.io/srns/},
timestamp = {2021-06-26T11:13:26.000+0200},
title = {Scene Representation Networks: Continuous 3D-Structure-Aware Neural
Scene Representations},
url = {http://arxiv.org/abs/1906.01618},
year = 2019
}