Data curation for foundation model training

Machine learning research has historically focused on algorithmic improvements, with better training methods driving innovation. Given that the amount of data available for training these models was often limited, research aimed on improving the way these relatively small amounts of data could be used. More recently, this focus has shifted from iteration on the algorithms to iteration on the data itself. Techniques such as data filtering, reannotation and data mixing, among many others, are often used to improve upon the data itself. In this work, we will examine ways to increase the quality of datasets in image-text and language domains, with a focus on dataset curation and filtering. Given the amount of readily available data on the web, these techniques can be reliably applied as methods to increase the downstream performance of models, with the improvement stemming directly from the higher quality datasets they were trained on. We will also examine dataset curation from the view of synthetic dataset generation in the domain of language model fine-tuning. These synthetic datasets can be used to distill reasoning capabilities from large, high-performing reasoning models to smaller, more compact ones, improving their usability and reducing inference costs.

Name thesis:degree_name

Doctor of Philosophy

Discipline thesis:degree_discipline

Electrical and Computer Engineering

Grantor

The University of Texas at Austin

Year dc:date.issued

2025

Author dc:creator

• Smyrnis, Georgios

Advisor dc:contributor.advisor

• Dimakis, Alexandros G.

Committee members dc:contributor.committeemember

• Ludwig Schmidt
• Sujay Sanghavi
• Sanjay Shakkottai
• Jonathan Tamir

• Foundation models
• Dataset curation
• Image-text models
• Language models

Language dc:language.iso

English

Identifier URI

https://doi.org/10.26153/tsw/61915

OAI identifier oai:identifier

oai:repositories.lib.utexas.edu:2152/134593