NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video
J. Sun, Y. Xie, L. Chen, X. Zhou, and H. Bao. (2021)cite arxiv:2104.00681Comment: Accepted to CVPR 2021 as Oral Presentation. Project page: https://zju3dv.github.io/neuralrecon/.
Abstract
We present a novel framework named NeuralRecon for real-time 3D scene
reconstruction from a monocular video. Unlike previous methods that estimate
single-view depth maps separately on each key-frame and fuse them later, we
propose to directly reconstruct local surfaces represented as sparse TSDF
volumes for each video fragment sequentially by a neural network. A
learning-based TSDF fusion module based on gated recurrent units is used to
guide the network to fuse features from previous fragments. This design allows
the network to capture local smoothness prior and global shape prior of 3D
surfaces when sequentially reconstructing the surfaces, resulting in accurate,
coherent, and real-time surface reconstruction. The experiments on ScanNet and
7-Scenes datasets show that our system outperforms state-of-the-art methods in
terms of both accuracy and speed. To the best of our knowledge, this is the
first learning-based system that is able to reconstruct dense coherent 3D
geometry in real-time.
%0 Generic
%1 sun2021neuralrecon
%A Sun, Jiaming
%A Xie, Yiming
%A Chen, Linghao
%A Zhou, Xiaowei
%A Bao, Hujun
%D 2021
%K 3d_reconstruction cvpr21 deeplearning neural_reconstruction
%T NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video
%U http://arxiv.org/abs/2104.00681
%X We present a novel framework named NeuralRecon for real-time 3D scene
reconstruction from a monocular video. Unlike previous methods that estimate
single-view depth maps separately on each key-frame and fuse them later, we
propose to directly reconstruct local surfaces represented as sparse TSDF
volumes for each video fragment sequentially by a neural network. A
learning-based TSDF fusion module based on gated recurrent units is used to
guide the network to fuse features from previous fragments. This design allows
the network to capture local smoothness prior and global shape prior of 3D
surfaces when sequentially reconstructing the surfaces, resulting in accurate,
coherent, and real-time surface reconstruction. The experiments on ScanNet and
7-Scenes datasets show that our system outperforms state-of-the-art methods in
terms of both accuracy and speed. To the best of our knowledge, this is the
first learning-based system that is able to reconstruct dense coherent 3D
geometry in real-time.
@misc{sun2021neuralrecon,
abstract = {We present a novel framework named NeuralRecon for real-time 3D scene
reconstruction from a monocular video. Unlike previous methods that estimate
single-view depth maps separately on each key-frame and fuse them later, we
propose to directly reconstruct local surfaces represented as sparse TSDF
volumes for each video fragment sequentially by a neural network. A
learning-based TSDF fusion module based on gated recurrent units is used to
guide the network to fuse features from previous fragments. This design allows
the network to capture local smoothness prior and global shape prior of 3D
surfaces when sequentially reconstructing the surfaces, resulting in accurate,
coherent, and real-time surface reconstruction. The experiments on ScanNet and
7-Scenes datasets show that our system outperforms state-of-the-art methods in
terms of both accuracy and speed. To the best of our knowledge, this is the
first learning-based system that is able to reconstruct dense coherent 3D
geometry in real-time.},
added-at = {2021-06-26T11:02:55.000+0200},
author = {Sun, Jiaming and Xie, Yiming and Chen, Linghao and Zhou, Xiaowei and Bao, Hujun},
biburl = {https://www.bibsonomy.org/bibtex/2ee9c52676fffad702f37276cd0869cf5/shuncheng.wu},
description = {https://arxiv.org/pdf/2104.00681.pdf},
interhash = {7da2e4514b16bd6653f5f564d1efbfcf},
intrahash = {ee9c52676fffad702f37276cd0869cf5},
keywords = {3d_reconstruction cvpr21 deeplearning neural_reconstruction},
note = {cite arxiv:2104.00681Comment: Accepted to CVPR 2021 as Oral Presentation. Project page: https://zju3dv.github.io/neuralrecon/},
timestamp = {2021-06-26T11:02:55.000+0200},
title = {NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video},
url = {http://arxiv.org/abs/2104.00681},
year = 2021
}