Efficient and effective learning of text representations

Text representation learning has played a pivotal role in enabling a wide range of Natural Language Processing (NLP) tasks that involve the processing, analysis, and utilization of human-generated textual data. These representations are typically obtained by transforming raw texts into vectors via deep neural networks (e.g., Transformers). Recent advances in Large Language Models (LLMs) have demonstrated the great potential of learning generic text representations applicable across a wide spectrum of applications. This success is underpinned by two critical factors: (1) the utilization of extensive text data for training LLMs in both pretraining and fine-tuning, and (2) the scaling up of LLMs to encompass tens or even hundreds of billions of parameters. Consequently, training LLMs entails substantial costs, including the acquisition of massive labeled data and the infrastructure required to support these large models. Motivated by these challenges, my doctoral research seeks to develop efficient and effective methods for learning text representations, spanning several key subtopics: 1. Leveraging the spherical space for text representation learning (Chapter 2). While the conventional choice for representation space is Euclidean, the non-Euclidean spherical space exhibits superior abilities to capture semantic correlations through directional similarity. My work focuses on self-supervised techniques that harness the spherical representation space for text representation learning. 2. Discovering topical structures with spherical text representations (Chapter 3). Based on the text representations learned in the spherical space, I have developed methods that automatically discover topical structures from a given corpus by jointly modeling topical and textual semantics. 3. Generating training data with LLMs for label-efficient natural language understanding (NLU) (Chapter 4). Achieving robust performance on NLU tasks typically necessitates a substantial number of human-annotated training samples for fine-tuning pretrained text representations. To mitigate the demands of human labeling, I have developed a new paradigm that employs LLMs as training data generators to replace the human annotation process. These endeavors collectively contribute to the more efficient and effective learning of text representations, addressing the challenges posed by the resource-intensive nature of training and using LLMs.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Meng, Yu

Contributors dc:contributor

• Han, Jiawei
• Abdelzaher, Tarek
• Tong, Hanghang
• Zettlemoyer, Luke

• Representation Learning
• Natural Language Processing
• Text Mining

Statement dc:rights

• Copyright 2023 Yu Meng

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/121956

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F121956