{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/129717"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/129717","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Improving neuron-level interpretability with white-box language models","abstract":"Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2027-05-01","abstract_html":"Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2027-05-01","abstract_has_math":false,"creators":["Bai, Hao"],"institution":"University of Illinois Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Jiang, Nan"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025-04-24","date_published":"2025-04-24","updated_at":"2026-07-22T22:25:05Z","subjects":["Neuron-level interpretability","language models","representation learning"],"languages":["en","eng"],"rights":["Copyright 2025 Hao Bai"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/129717","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Jiang, Nan"]},{"key":"dc:creator","label":"Author","values":["Bai, Hao"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2025-04-24","2025-05"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Neuron-level interpretability","language models","representation learning"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2025 Hao Bai"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/129717"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2027-05-01","The student, Hao Bai, accepted the attached license on 2025-04-24 at 10:25.","The student, Hao Bai, submitted this Thesis for approval on 2025-04-24 at 10:31.","This Thesis was approved for publication on 2025-04-24 at 17:11.","DSpace SAF Submission Ingestion Package generated from Vireo submission #21944 on 2025-10-19 at 19:53:52","Neurons in auto-regressive language models like GPT-2 can be interpreted by analyzing their activation patterns. Recent studies have shown that techniques such as dictionary learning, a form of post-hoc sparse coding, enhance this neuron-level interpretability. In our research, we are driven by the goal to fundamentally improve neural network interpretability by embedding sparse coding directly within the model architecture, rather than applying it as an afterthought. In our study, we introduce a white-box transformer-like architecture named Coding RAte TransformEr (crate), explicitly engineered to capture sparse, lowdimensional structures within data distributions. Our comprehensive experiments showcase significant improvements (up to 103% relative improvement) in neuron-level interpretability across a variety of evaluation metrics. Detailed investigations confirm that this enhanced interpretability is steady across different layers irrespective of the model size, underlining crate’s robust performance in enhancing neural network interpretability. Further analysis shows that crate’s increased interpretability comes from its enhanced ability to consistently and distinctively activate on relevant tokens. These findings point towards a promising direction for creating white-box foundation models that excel in neuron-level interpretation."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Improving neuron-level interpretability with white-box language models"]}]}],"canonical_facts":{"dc:contributor":["Jiang, Nan"],"dc:creator":["Bai, Hao"],"dc:date":["2025-04-24","2025-05"],"dc:description":["Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2027-05-01","The student, Hao Bai, accepted the attached license on 2025-04-24 at 10:25.","The student, Hao Bai, submitted this Thesis for approval on 2025-04-24 at 10:31.","This Thesis was approved for publication on 2025-04-24 at 17:11.","DSpace SAF Submission Ingestion Package generated from Vireo submission #21944 on 2025-10-19 at 19:53:52","Neurons in auto-regressive language models like GPT-2 can be interpreted by analyzing their activation patterns. Recent studies have shown that techniques such as dictionary learning, a form of post-hoc sparse coding, enhance this neuron-level interpretability. In our research, we are driven by the goal to fundamentally improve neural network interpretability by embedding sparse coding directly within the model architecture, rather than applying it as an afterthought. In our study, we introduce a white-box transformer-like architecture named Coding RAte TransformEr (crate), explicitly engineered to capture sparse, lowdimensional structures within data distributions. Our comprehensive experiments showcase significant improvements (up to 103% relative improvement) in neuron-level interpretability across a variety of evaluation metrics. Detailed investigations confirm that this enhanced interpretability is steady across different layers irrespective of the model size, underlining crate’s robust performance in enhancing neural network interpretability. Further analysis shows that crate’s increased interpretability comes from its enhanced ability to consistently and distinctively activate on relevant tokens. These findings point towards a promising direction for creating white-box foundation models that excel in neuron-level interpretation."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/129717"],"dc:language":["en","eng"],"dc:rights":["Copyright 2025 Hao Bai"],"dc:subject":["Neuron-level interpretability","language models","representation learning"],"dc:title":["Improving neuron-level interpretability with white-box language models"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:05Z"}