Structured prediction for compute efficient and high accuracy NLP

Structured prediction in machine learning focuses on mapping a sequence of inputs to a sequence of outputs within a vast output space, with interconnected predictions, offering simplicity and speed while enhancing contextual understanding in NLP tasks. In this dissertation, we revisit the applicability of structured prediction in modern, intricate NLP applications. We introduce a structured prediction-based approach for extracting linguistic knowledge from pre-trained encoder-only language models, and demonstrate the effectiveness of the extracted knowledge in enhancing translation quality of encoder-decoder models. We showcase the efficacy of well-designed, simple structured prediction-based sequence labelling in handling complex entity linking with large entity vocabularies. Our proposed method, SpEL, not only simplifies and accelerates the process but also achieves state-of-the-art results on a prominent entity linking benchmark dataset. Furthermore, we investigate Entity Retrieval, the application of our structured prediction-based entity linking framework as an alternative strategy to prevalent dense retrieval methods in retrieval-augmented question answering, particularly for factual questions about the real world. Our research underscores structured prediction as a compelling approach for modelling complex NLP tasks, particularly when prioritizing computational efficiency and high accuracy. We conclude the dissertation with a review of additional contributions that either diverge from the primary focus or involve shared authorship, even if they pertain to the central theme of the dissertation.