Cloud-assisted real-time free viewpoint video rendering and streaming System

Free Viewpoint Video (FVV) is an emerging type of video which allows user to choose viewpoint freely in three-dimensional scenes. Depth-image-based Rendering (DIBR) is a common method to generate FVV using both texture and depth information. However, FVV rendering is more time-consuming than the original video since it has higher computational complexity. In order to make FVV rendering in real-time, a cloud-assisted system is proposed, which leverages cloud and distributed computing. In addition, we use multithread programming to take full advantage of cloud resources. As a result, by deploying our system on the WestGrid cluster, the FVV generation speed can be over 30 fps. Furthermore, to achieve the optimal trade-off between economic cost and user experience, we formulate and build mathematical models for the cloud-based FVV rendering and streaming system. Based on that, dynamic resource allocation algorithms are designed, which can provide the optimal resource allocation scheme according to users’ requests. The performance of the system is demonstrated by various experiments. To the best of our knowledge, this is the first cloud-assisted real-time FVV rendering and streaming system.