Structured outdoor architecture reconstruction through RGB image

The automated construction of structured geometry is crucial for the next technological revolution in broader domains such as digital mapping, construction, robotics, augmented reality, and more. However, constructing structured geometry (e.g. polygonal shape) is still a challenging problem without compelling solutions. This thesis tackles a structured reconstruction problem towards such revolution, whose task is to infer an outdoor building architecture as a 2D planar graph from a single RGB image. We propose a novel pipeline with two modules: 1) Construction module learns to select a subset of all possible building edges to reconstruct a planar graph with a novel convolutional Message Passing Neural Network (Conv-MPN); and 2) Refinement module starts from the result of the construction module and explores better models by iteratively refining and classifying their correctness. Concretely, our method vectorizes the structures with semantically meaningful corners, edges and regions, which benefit many downstream applications such as VR/AR and image-editing. Qualitative and quantitative evaluations over two thousand buildings demonstrate that our approach makes significant improvements over all the existing methods. Our modules are effective in other structured geometry analysis tasks. For example, Conv-MPN is already used for the floorplan generation and piecewise planar reconstruction from a single image.