Motion modeling and segmentation in compressed video with applications

This Ph.D. thesis offers a perspective of the theory, applications, and implementation of motion modeling and segmentation in compressed video. The research effort is devoted to retrieving object motion information from compressed video and producing images with desired spatial and temporal resolutions. On the theoretical side, this research project attempts to model motion and retrieve substantial information about objects in 3D real-life scenes from compressed 2D images, and develop techniques to reconstruct an image that fits into the same perspective of the sequence with desired temporal and spatial resolutions, which might or might not conform to those of the original sequence. From an application perspective, three types of scenarios are then classified: 1) frame prediction, 2) frame interpolation and 3) frame resizing. Scenario 1) involves temporal causal processing where video data from the past is used, while scenario 2) is temporal, noncausal processing where both past and future video data are exploited. A variety of video applications can be found in either of these two scenarios, e.g., predictive decoding for delay reduction, whole-frame error concealment, playout buffer control for video streaming, frame rate up-conversion, and so on. Scenario 3) is spatial processing for producing a frame with desired spatial resolution, e.g., super-resolution reconstruction of compressed video. In addition, the techniques developed to detect and track objects in compressed video can benefit many content-based video applications, such as effectively indexing, searching and categorising video sequences. This is of particular interest as broadband internet access is now more widely available. State-of-the-art block-based video codecs are targeted for the implementation and evaluation of the effectiveness of the proposed system.