{"id":{"repo_id":"mit","oai_identifier":"oai:dspace.mit.edu:1721.1/157345"},"canonical_url":"https://search.dev.ndltd.org/etd/mit/oai:dspace.mit.edu:1721.1/157345","repository":{"repo_id":"mit","name":"MIT","base_url":"https://dspace.mit.edu/oai/request"},"display":{"title":"Comparing Parameter Efficient Finetuning Techniques (PEFT) using Datamodels","abstract":"Advances in machine learning, particularly through algorithmic innovations and large datasets, have led to models with hundreds of billions of parameters. Deploying these models is challenging and costly, especially due to the extensive finetuning required. Parameter-efficient finetuning techniques (PEFT) have been proposed to address this issue by significantly reducing the number of trainable parameters, achieving comparable results to full-parameter finetuning. Despite widespread adoption, PEFT methods are often used interchangeably without considering their qualitative differences and performance under various data distributions. This thesis extensively compares three PEFT methods: LoRA, BitFit, and (IA)³, using the ModelDiff framework to identify and apply data interventions. Our analysis reveals that the performance of these methods varies widely with different interventions, with BitFit showing the most variance, while LoRA and (IA)³ demonstrate greater resilience. This study informs the selection and optimization of PEFT techniques based on specific NLP task requirements, balancing performance, computational efficiency, and robustness to text variations.","abstract_html":"Advances in machine learning, particularly through algorithmic innovations and large datasets, have led to models with hundreds of billions of parameters. Deploying these models is challenging and costly, especially due to the extensive finetuning required. Parameter-efficient finetuning techniques (PEFT) have been proposed to address this issue by significantly reducing the number of trainable parameters, achieving comparable results to full-parameter finetuning. Despite widespread adoption, PEFT methods are often used interchangeably without considering their qualitative differences and performance under various data distributions. This thesis extensively compares three PEFT methods: LoRA, BitFit, and (IA)³, using the ModelDiff framework to identify and apply data interventions. Our analysis reveals that the performance of these methods varies widely with different interventions, with BitFit showing the most variance, while LoRA and (IA)³ demonstrate greater resilience. This study informs the selection and optimization of PEFT techniques based on specific NLP task requirements, balancing performance, computational efficiency, and robustness to text variations.","abstract_has_math":false,"creators":["Chamdal, Harshal"],"institution":"Massachusetts Institute of Technology","degree_name":"Master","degree_level":null,"degree_discipline":null,"degree_department":"Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science","school":null,"contributors":[],"advisors":["Mądry, Aleksander"],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024-05","date_published":"2024-05","updated_at":"2026-07-22T22:22:05Z","subjects":[],"languages":[],"rights":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"],"rights_urls":["https://rightsstatements.org/page/InC-EDU/1.0/"],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/1721.1/157345","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Mądry, Aleksander"]},{"key":"dc:contributor.department","label":"Department","values":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"]},{"key":"dc:creator","label":"Author","values":["Chamdal, Harshal"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2024-10-16T17:44:13Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2024-10-16T17:44:13Z"]},{"key":"dc:date.issued","label":"Date","values":["2024-05"]},{"key":"dc:publisher","label":"Institution","values":["Massachusetts Institute of Technology"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master","Master of Engineering in Electrical Engineering and Computer Science"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:rights","label":"Dc Rights","values":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"]},{"key":"dc:rights.uri","label":"Rights URI","values":["https://rightsstatements.org/page/InC-EDU/1.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/1721.1/157345"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Advances in machine learning, particularly through algorithmic innovations and large datasets, have led to models with hundreds of billions of parameters. Deploying these models is challenging and costly, especially due to the extensive finetuning required. Parameter-efficient finetuning techniques (PEFT) have been proposed to address this issue by significantly reducing the number of trainable parameters, achieving comparable results to full-parameter finetuning. Despite widespread adoption, PEFT methods are often used interchangeably without considering their qualitative differences and performance under various data distributions. This thesis extensively compares three PEFT methods: LoRA, BitFit, and (IA)³, using the ModelDiff framework to identify and apply data interventions. Our analysis reveals that the performance of these methods varies widely with different interventions, with BitFit showing the most variance, while LoRA and (IA)³ demonstrate greater resilience. This study informs the selection and optimization of PEFT techniques based on specific NLP task requirements, balancing performance, computational efficiency, and robustness to text variations."]},{"key":"dc:description.degree","label":"Dc Description Degree","values":["M.Eng."]},{"key":"dc:title","label":"Title","values":["Comparing Parameter Efficient Finetuning Techniques (PEFT) using Datamodels"]}]}],"canonical_facts":{"dc:contributor.advisor":["Mądry, Aleksander"],"dc:contributor.department":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"],"dc:creator":["Chamdal, Harshal"],"dc:date.accessioned":["2024-10-16T17:44:13Z"],"dc:date.available":["2024-10-16T17:44:13Z"],"dc:date.issued":["2024-05"],"dc:description.abstract":["Advances in machine learning, particularly through algorithmic innovations and large datasets, have led to models with hundreds of billions of parameters. Deploying these models is challenging and costly, especially due to the extensive finetuning required. Parameter-efficient finetuning techniques (PEFT) have been proposed to address this issue by significantly reducing the number of trainable parameters, achieving comparable results to full-parameter finetuning. Despite widespread adoption, PEFT methods are often used interchangeably without considering their qualitative differences and performance under various data distributions. This thesis extensively compares three PEFT methods: LoRA, BitFit, and (IA)³, using the ModelDiff framework to identify and apply data interventions. Our analysis reveals that the performance of these methods varies widely with different interventions, with BitFit showing the most variance, while LoRA and (IA)³ demonstrate greater resilience. This study informs the selection and optimization of PEFT techniques based on specific NLP task requirements, balancing performance, computational efficiency, and robustness to text variations."],"dc:description.degree":["M.Eng."],"dc:identifier.uri":["https://hdl.handle.net/1721.1/157345"],"dc:publisher":["Massachusetts Institute of Technology"],"dc:rights":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"],"dc:rights.uri":["https://rightsstatements.org/page/InC-EDU/1.0/"],"dc:title":["Comparing Parameter Efficient Finetuning Techniques (PEFT) using Datamodels"],"dc:type":["Thesis"],"thesis:degree_name":["Master","Master of Engineering in Electrical Engineering and Computer Science"]},"updated_at":"2026-07-22T22:22:05Z"}