Mirror3D: Depth refinement for mirror surfaces

Despite advances in depth sensing and 3D reconstruction, mirror surfaces remain a significant source of errors. In existing 3D reconstruction, mirror surfaces are either missing or have incorrect depth values. These artifacts degrade the performance of computer vision tasks that rely on high-quality 3D reconstruction. This thesis proposes the task of 3D mirror plane prediction given an RGB or RGB-D image. The 3D mirror plane prediction task aims to locate a mirror plane in 3D indoor scenes so that we can refine the erroneous mirror regions in 3D environments. To accomplish this task, we first create the Mirror3D dataset, the first 3D mirror dataset which corrects mirror depth from three public RGB-D datasets (Matterport3D, NYUv2, and ScanNet) and contains 7,011 mirror instance masks and 3D plane annotations. With the Mirror3D dataset, we then develop a neural network architecture --- Mirror3DNet to refine the mirror depth values in both estimated depth maps and raw sensor depth maps. The key idea of Mirror3DNet is to estimate 3D mirror plane parameters based on RGB input and the mirror's surrounding depth context. We establish benchmarks for RGB and RGB-D based 3D mirror plane prediction by evaluating existing work on depth completion and depth estimation. Our qualitative and quantitative evaluation results show that depth maps that are assumed as ground truth in the current RGB-D datasets can significantly misrepresent the depth prediction performance. Finally, we show that the Mirror3D architecture we propose Mirror3DNet can effectively refine both estimated depth maps and raw depth maps from depth sensors.