A neural network for monocular point cloud estimation of human objects

This work introduces a neural network for estimating the detailed 3D structure of the foreground human in a single RGB image, which can capture geometry details like cloth wrinkles. The outputs of the network can potentially be fused together for various applications, such as tele-presence and 3D scanning. The neural network is designed with the insight of separating the 3D shape into a smooth base shape and a residual detail shape such that each component is trained by an independent sub-network. For training the neural network, we also capture our own data and apply non-rigid registration techniques for post-processing. Furthermore, we introduce a novel refinement technique based on non-rigid alignment to improve the temporal consistency of the final results. Quantitative comparison with fused ground truth captured by real depth cameras and qualitative examples on test images demonstrate the strength of the proposed method.