{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/124446"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/124446","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Exploring efficiency improvements to distillation of language models for code","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","abstract_has_math":false,"creators":["Karthikeyan, Ajaykrishna"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Zhang, Lingming"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024-05","date_published":"2024-05","updated_at":"2026-07-22T22:25:00Z","subjects":["Machine Learning","Language Models For Code","Distillation","Fine-tuning","Efficiency","Subsampling"],"languages":["en","eng"],"rights":["Copyright 2024 Ajaykrishna Karthikeyan"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/124446","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Zhang, Lingming"]},{"key":"dc:creator","label":"Author","values":["Karthikeyan, Ajaykrishna"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2024-05","2024-05-01"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Machine Learning","Language Models For Code","Distillation","Fine-tuning","Efficiency","Subsampling"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2024 Ajaykrishna Karthikeyan"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/124446"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","The student, Ajaykrishna Karthikeyan, accepted the attached license on 2024-04-30 at 21:54.","The student, Ajaykrishna Karthikeyan, submitted this Thesis for approval on 2024-04-30 at 22:07.","This Thesis was approved for publication on 2024-05-01 at 11:43.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20729 on 2024-09-16 at 00:37:42","Large Language Models (LLMs) trained on code have demonstrated great success in the code domain across various tasks including code generation, repair, test generation, and more. These models are increasingly being adopted in both industry for practical software development and in academia to address challenges in software engineering that previously relied on heuristic-based methods or conventional machine learning techniques. While prompting can be an easy way to deploy these models for certain tasks, fine-tuning them for specific tasks often yields more reliable results. Therefore fine-tuning is widely prevalent in the industry and is still used for several tasks in academia. However, with the increasing size of these models, a major drawback of fine-tuning is the high cost, arising primarily from the extensive GPU compute time required. This compute time is especially large when we consider the full development and debugging phase of fine-tuning these models. Therefore, in this study, we aim to reduce these costs by reducing the time needed for fine-tuning while striving to minimally impact predictive performance. We primarily explore data subsampling techniques that can help reduce the size of the fine-tuning dataset thereby reducing associated costs. We specifically focus on fine-tuning within the context of model distillation. Previous research has utilized functional correctness-based methods for data sub-sampling, which can be effective for standalone programming tasks. However, many real-world coding tasks involve complexities such as setting up environments or integrating external resources, making evaluating functional correctness infeasible. Our work explores alternative sampling-based techniques that rely only on the probabilistic outputs of models instead of code execution. Through extensive empirical evaluation on an MBPP-based dataset, we show the feasibility of reducing fine-tuning time while maintaining performance. More specifically, we demonstrate that our methodology can achieve a 37% reduction in fine-tuning time with negligible impact on predictive performance, and in some instances, up to a 75% reduction without any loss in performance."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Exploring efficiency improvements to distillation of language models for code"]}]}],"canonical_facts":{"dc:contributor":["Zhang, Lingming"],"dc:creator":["Karthikeyan, Ajaykrishna"],"dc:date":["2024-05","2024-05-01"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","The student, Ajaykrishna Karthikeyan, accepted the attached license on 2024-04-30 at 21:54.","The student, Ajaykrishna Karthikeyan, submitted this Thesis for approval on 2024-04-30 at 22:07.","This Thesis was approved for publication on 2024-05-01 at 11:43.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20729 on 2024-09-16 at 00:37:42","Large Language Models (LLMs) trained on code have demonstrated great success in the code domain across various tasks including code generation, repair, test generation, and more. These models are increasingly being adopted in both industry for practical software development and in academia to address challenges in software engineering that previously relied on heuristic-based methods or conventional machine learning techniques. While prompting can be an easy way to deploy these models for certain tasks, fine-tuning them for specific tasks often yields more reliable results. Therefore fine-tuning is widely prevalent in the industry and is still used for several tasks in academia. However, with the increasing size of these models, a major drawback of fine-tuning is the high cost, arising primarily from the extensive GPU compute time required. This compute time is especially large when we consider the full development and debugging phase of fine-tuning these models. Therefore, in this study, we aim to reduce these costs by reducing the time needed for fine-tuning while striving to minimally impact predictive performance. We primarily explore data subsampling techniques that can help reduce the size of the fine-tuning dataset thereby reducing associated costs. We specifically focus on fine-tuning within the context of model distillation. Previous research has utilized functional correctness-based methods for data sub-sampling, which can be effective for standalone programming tasks. However, many real-world coding tasks involve complexities such as setting up environments or integrating external resources, making evaluating functional correctness infeasible. Our work explores alternative sampling-based techniques that rely only on the probabilistic outputs of models instead of code execution. Through extensive empirical evaluation on an MBPP-based dataset, we show the feasibility of reducing fine-tuning time while maintaining performance. More specifically, we demonstrate that our methodology can achieve a 37% reduction in fine-tuning time with negligible impact on predictive performance, and in some instances, up to a 75% reduction without any loss in performance."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/124446"],"dc:language":["en","eng"],"dc:rights":["Copyright 2024 Ajaykrishna Karthikeyan"],"dc:subject":["Machine Learning","Language Models For Code","Distillation","Fine-tuning","Efficiency","Subsampling"],"dc:title":["Exploring efficiency improvements to distillation of language models for code"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:00Z"}