Multimodal guidance for medical image classification

Medical imaging is a cornerstone of therapy and diagnosis in modern medicine. However, the choice of imaging modality for a particular theranostic task typically involves trade-offs between the feasibility of using a particular modality (e.g., short wait times, low cost, fast acquisition, reduced radiation/invasiveness) and the expected performance on a clinical task (e.g., diagnostic accuracy, efficacy of treatment planning and guidance). The goal of this thesis is to examine the ability to apply the knowledge learned from the less feasible but better-performing (superior) modality to guide the utilization of the more-feasible yet under-performing (inferior) modality and steer it towards improved performance. To this end, we develop a lightweight guidance model – an autoencoder-like deep neural network – that learns a mapping from the latent representation of the inferior modality to the latent representation of its superior counterpart. With the incorporation of this model in the classification framework of the inferior modality, we aim to compensate for the absence of the superior modality during inference time. We focus on the application of deep learning for image-based diagnosis and examine the advantages of our method in the context of two clinical applications: multi-task skin lesion classification from clinical and dermoscopic images and brain tumor classification from multi-sequence magnetic resonance imaging (MRI) and histopathology images. For both these scenarios, we show a boost in the diagnostic performance of the inferior modality without requiring the superior modality. Furthermore, in the case of brain tumor classification, our method outperforms the model trained on the superior modality while producing comparable results to the model that uses both modalities during inference.