Mirror3D: Depth refinement for mirror surfaces

Resource type
Thesis type
(Thesis) M.Sc.
Date created
Author: Tan, Jiaqi
Despite advances in depth sensing and 3D reconstruction, mirror surfaces remain a significant source of errors. In existing 3D reconstruction, mirror surfaces are either missing or have incorrect depth values. These artifacts degrade the performance of computer vision tasks that rely on high-quality 3D reconstruction. This thesis proposes the task of 3D mirror plane prediction given an RGB or RGB-D image. The 3D mirror plane prediction task aims to locate a mirror plane in 3D indoor scenes so that we can refine the erroneous mirror regions in 3D environments. To accomplish this task, we first create the Mirror3D dataset, the first 3D mirror dataset which corrects mirror depth from three public RGB-D datasets (Matterport3D, NYUv2, and ScanNet) and contains 7,011 mirror instance masks and 3D plane annotations. With the Mirror3D dataset, we then develop a neural network architecture --- Mirror3DNet to refine the mirror depth values in both estimated depth maps and raw sensor depth maps. The key idea of Mirror3DNet is to estimate 3D mirror plane parameters based on RGB input and the mirror's surrounding depth context. We establish benchmarks for RGB and RGB-D based 3D mirror plane prediction by evaluating existing work on depth completion and depth estimation. Our qualitative and quantitative evaluation results show that depth maps that are assumed as ground truth in the current RGB-D datasets can significantly misrepresent the depth prediction performance. Finally, we show that the Mirror3D architecture we propose Mirror3DNet can effectively refine both estimated depth maps and raw depth maps from depth sensors.
58 pages.
Copyright statement
Copyright is held by the author(s).
This thesis may be printed or downloaded for non-commercial research and scholarly purposes.
Supervisor or Senior Supervisor
Thesis advisor: Savva, Manolis
Thesis advisor: Chang, Angel
Member of collection
Download file Size
etd21798.pdf 18.92 MB

Views & downloads - as of June 2023

Views: 11
Downloads: 0