Exploiting side information and scalability in compressed sensing and deep learning

There is a tremendous demand for increasingly efficient ways of both capturing and processing high-dimensional datasets of large size. When capturing such datasets, a promising recent trend has developed based on the recognition that, many high-dimensional datasets have low-dimensional structures. For example, the notion of sparsity is a requisite in the compressed sensing (CS) field, which allows for accurate signal reconstruction from sub-Nyquist sampled measurements given certain conditions. When processing such datasets, the recently developed deep learning is a powerful tool, able to extract high-level and complex abstractions from massive amounts of data. CS has a wide range of applications that include imaging, radar and many more. Much effort has been put on developing more accurate and efficient reconstruction algorithms. In this thesis, first, we are interested in how to incorporate the side information into CS reconstruction when there is an initial estimation of the sparse signal available from other sources. Rigorous theoretical analysis was proposed for the first time in this field. Sufficient number of measurements is required for accurate CS reconstruction. We may have to wait for a long time to do the reconstruction until we receive enough measurements, which could incur undesired delays. Moreover, state-of-the-art CS reconstruction algorithms are still inefficient for signals of large size, e.g., images. Inspired by the multi-resolution or scalable reconstruction in multimedia transmission, such as JPEG 2000 and H.264/SVC, in the second part of this thesis, we analyzed scalable CS reconstruction problem and proposed to reconstruct a low-resolution signal if the number of measurements is too small. Deep learning or deep neural networks (DNNs) has evolved into the state-of-the-art technique for many artificial intelligence tasks including computer vision, speech recognition and natural language processing. However, DNNs generally involve many layers with millions of parameters, making them difficult to be deployed and updated on devices with limited resources such as mobile phones and other smart embedded systems. Moreover, if the DNN needs to be updated, usually via wireless communications, downloading the large amount of network parameters will cause excessive delay. In the final part of this thesis, we propose a scalable representation of the network parameters, so that different applications can select the most suitable bit rate of the network based on their own storage constraints.