Knowledge representation and behavior understanding with pre-trained language models

The advent of pre-trained language models has marked a significant milestone in the realm of computational linguistics and data mining, showcasing remarkable performances across diverse domains. These models, endowed with vast training corpora and formidable capacity for semantic knowledge representation, have transformed the landscape of different downstream applications. Among the various applications, the automatic construction and completion of taxonomies have shown to be beneficial in enhancing numerous downstream tasks and minimizing human labor in domain-specific taxonomy development. In this work, we first introduce a taxonomy completion framework that effectively leverages pre-trained language models to extract structural and semantic information from the existing taxonomy to significantly boost the performance of current taxonomy expansion and completion frameworks. On the other hand, even though the performances of pre-trained language models are very high in many datasets, the implications of data contamination during the pre-training stage of language models are still unclear in the current literature. Given their demonstrated prowess in enhancing task performance across diverse downstream applications, concerns arise regarding the authenticity of these capabilities, potentially inflated by the inadvertent inclusion of evaluation datasets within pre-training corpora. Through meticulous experimental investigation, this study endeavors to elucidate the effects of data contamination, emphasizing the imperative for more precise definitions and stringent methodologies to fortify LLMs against such vulnerabilities.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Jiang, Minhao

Contributors dc:contributor

• Han, Jiawei

• Pre-trained Language Models
• Knowledge Representation
• Data Contamination

Statement dc:rights

• Copyright 2024 Minhao Jiang

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124388

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124388