Automated Finetuning via Sparse Autoencoders

Currently, the field of interpretability is traditionally confined to diagnostics. However, this thesis presents a novel method using interpretability in sparse autoencoders to achieve better performance in small models via instruction finetuning. Specifically, we present UnderstandTune, an autonomous method for assembling high-quality instruction finetuning datasets with minimal human intervention, requiring only concise task descriptions rather than evaluation dataset distributions. Our empirical evaluations show that UnderstandTune consistently outperforms uninformed finetuning baselines across multiple benchmarks. Complementing this, Lalon introduces a mixture-of-informed-experts (MoIE) architecture that routes queries to specialized models independently finetuned via UnderstandTune. This modular approach achieves competitive performance against larger monolithic models in specialized domains, while utilizing fewer parameters, training examples, and computational resources. The framework’s modularity enables independent optimization of components from sparse autoencoders to MoIE routing mechanisms. This research demonstrates how interpretability can be used to enhance performance through intelligent data curation and suggests a new paradigm where interpretability and efficiency reinforce each other toward more capable, resource-efficient AI systems.

Name thesis:degree_name

Master

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2025

Author dc:creator

• Sivakumar, Ragulan

Advisor dc:contributor.advisor

• Berger, Bonnie

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright retained by author(s)

Licence dc:rights.uri

• https://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/163022

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/163022