Association knowledge in natural language learning

Association is an important feature of natural language originates from the human's cognitive ability to associate concepts. It is the key to unveiling the computational mechanism of natural language, which is closely related to the research of natural language processing (NLP). However, the current dominant learning paradigm, which is based on neural models that combine distributed representations with probabilistic modeling, demonstrates insufficient capabilities in modeling associations in natural language. To compensate for the deficiency, we mathematically formulate the concept of Association Knowledge as the joint distribution over the probabilities of instances and establish a general methodology to incorporate association knowledge into the training architecture of neural models. We delve into Association Knowledge through a series of case studies across various dimensions, including associations among types of knowledge, languages, instances and unstructured information. These case studies span both smaller neural models and large language models. Through our investigations, we demonstrate that explicitly integrating Association Knowledge into neural architectures markedly improves model performance and efficiency. This enhanced capability is evident in diverse scenarios, from improving event detection in lifelong learning settings and facilitating robust cross-lingual translations, to enhancing the detection of long-tail mentions and refining the updates in large language models. Collectively, our findings underscore the pivotal role of Association Knowledge in advancing the state of NLP by fostering more robust and knowledge-aware neural models.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Yu, Pengfei

Contributors dc:contributor

• Ji, Heng
• Han, Jiawei
• Hoiem, Derek
• Neubig, Graham
• Yih, Scott

• Natural Language Processing
• Association Modeling
• Information Extraction
• Knowledge Editing

Statement dc:rights

• Copyright 2024 Pengfei Yu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/127164

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F127164