Towards Fine-grained Multi-Attribute Control using Language Models

As we increasingly rely on powerful language models, ensuring their safe and effective operation necessitates extensive research in controllable text generation. Existing state-of-the-art language models struggle to generate the most accurate or desired output at the first attempt. Inspired by recent developments in self-correction in large language models and new reinforcement learning methods, we aim to train smaller language models as fine-grained editors, whereby they iteratively edit outputs to satisfy threshold constraints over multiple classifier-based attributes. In this thesis, I show a study of contextual offensive behavior of pretrained large language models and curate a high-quality dataset for toxicity detection. Next, I introduce a novel offline RL algorithm that can utilize arbitrary numeric scores as rewards during training to optimize any user-desired LM behavior by filtering out suboptimal data. Finally, I designed an offline RL framework, I propose a fine-grained multi-attribute controllability task, where the goal is to guide the language model to generate output sequences that satisfy user-defined threshold-based attribute constraints. The LM model can take multiple edits to reach the desired attributes. Experiments on both languages and proteins demonstrate the versatility and effectiveness of our approach.

Level thesis:degree_level

Doctoral

Department dc:contributor.department

Interactive Computing

Grantor dc:publisher

Georgia Institute of Technology

Year dc:date.issued

2024

Author dc:creator

• Baheti, Ashutosh

Advisors dc:contributor.advisor

• Riedl, Mark O.
• Ritter, Alan

Committee members dc:contributor.committeemember

• Batra, Dhruv
• Choudhury, Munmun De
• Sap, Maarten

• Reinforcement Learning
• Large Language Model
• Controlled Text Generation

Language dc:language.iso

en_US

Handle dc:identifier.uri

https://hdl.handle.net/1853/75278

OAI identifier oai:identifier

oai:repository.gatech.edu:1853/75278