Knowledge acquisition for natural language understanding

Large neural models pretrained on vast volumes of text have achieved remarkable success in various natural language processing tasks. However, these models may still face challenges in knowledge-intensive tasks due to their training methods, which typically focus on learning directly from raw texts and do not incorporate existing linguistic resources or structured domain knowledge. This thesis aims to develop methods to effectively incorporate external knowledge into existing neural models to enhance their performance. We propose three novel approaches that offer varying levels of explicitness for incorporating external knowledge into neural models, accommodating a wide range of use cases and providing great flexibility. The first approach involves incorporating various types of domain knowledge from multiple sources into language models using lightweight adapter modules. For each knowledge source of interest, we train an adapter module to capture the knowledge in a self-supervised way. The knowledge encoded in the adapters can then be combined for downstream tasks using fusion layers. This approach provides an easy-to-use, implicit way of incorporating external knowledge. The second approach involves utilizing a retrieval system to retrieve relevant passages from a knowledge base, which can then be used to enhance an output generation model. To train the retrieval components, we use a novel method for generating pseudo-labels to avoid the need for collecting costly gold-standard retrieval labels. This approach offers a more explicit way of accessing and using external knowledge than the adapter-based approach and provides greater interpretability. Additionally, new knowledge can typically be added to the knowledge base without updating any parameters of the neural component. The third approach involves using entity linking to extract the exact part of a knowledge graph that is relevant to the task at hand. We then utilize graph neural networks to incorporate the extracted subgraph into the existing neural model. This approach provides an even more explicit way of incorporating external knowledge, allowing for fine-grained control over what knowledge to incorporate and offering even more interpretability. We demonstrate the effectiveness of our proposed methods on various knowledge-intensive natural language processing tasks, including biomedical information extraction and knowledge-grounded dialog. We show that incorporating external knowledge can help overcome the difficulty of learning domain-specific knowledge and enhance the model's efficiency and interpretability. Our methods also allow for natural updates and additions of external knowledge, providing a flexible and scalable way of enhancing large neural language models. Overall, our methods achieve state-of-the-art results on many benchmarks.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Lai, Tuan Manh

Contributors dc:contributor

• Ji, Heng
• Zhai, ChengXiang
• Han, Jiawei
• Bui, Trung H

• Natural Language Processing
• Information Extraction
• Deep Learning
• Large Language Models

Statement dc:rights

• Copyright 2023 Tuan Lai

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/121297

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F121297