{"id":{"repo_id":"uoit","oai_identifier":"oai:ontariotechu.scholaris.ca:10155/1982"},"canonical_url":"https://search.dev.ndltd.org/etd/uoit/oai:ontariotechu.scholaris.ca:10155/1982","repository":{"repo_id":"uoit","name":"Ontario Institute of Technology","base_url":"https://ontariotechu.scholaris.ca/server/oai/request"},"display":{"title":"Towards searching for the best student in a Knowledge Distillation framework","abstract":"Knowledge Distillation (KD) enables the creation of compact student models that can retain much of the predictive power of larger teacher models. While these student models are invaluable for deployment in resource-constrained environments, the process of identifying an optimal student—balancing architecture and training hyperparameters—is often hindered by the extensive and computationally intensive search required. This thesis introduces the KD-Policy-Learning (KD-PL) framework, a novel approach designed to mitigate this challenge. KD-PL integrates an explicit caching mechanism for previously evaluated configurations and an adaptive proximity analysis module within a Reinforcement Learning (RL) agent. This agent performs joint Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) specifically for KD. By systematically avoiding redundant evaluations of identical configurations and accurately estimating the performance of similar, novel ones, our framework markedly improves search efficiency. Experimental results demonstrate that KD-PL significantly reduces the computational cost of discovering effective student models—achieving substantial wall-clock time savings (e.g., 70–93%) compared to strong baselines. This enhanced efficiency is particularly beneficial when working with large teacher models or under strict resource budgets, all while maintaining competitive student model accuracy.","abstract_html":"Knowledge Distillation (KD) enables the creation of compact student models that can retain much of the predictive power of larger teacher models. While these student models are invaluable for deployment in resource-constrained environments, the process of identifying an optimal student—balancing architecture and training hyperparameters—is often hindered by the extensive and computationally intensive search required. This thesis introduces the KD-Policy-Learning (KD-PL) framework, a novel approach designed to mitigate this challenge. KD-PL integrates an explicit caching mechanism for previously evaluated configurations and an adaptive proximity analysis module within a Reinforcement Learning (RL) agent. This agent performs joint Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) specifically for KD. By systematically avoiding redundant evaluations of identical configurations and accurately estimating the performance of similar, novel ones, our framework markedly improves search efficiency. Experimental results demonstrate that KD-PL significantly reduces the computational cost of discovering effective student models—achieving substantial wall-clock time savings (e.g., 70–93%) compared to strong baselines. This enhanced efficiency is particularly beneficial when working with large teacher models or under strict resource budgets, all while maintaining competitive student model accuracy.","abstract_has_math":false,"creators":["Bakos, Steve"],"institution":"University of Ontario Institute of Technology","degree_name":"Master of Science (MSc)","degree_level":null,"degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":[],"advisors":["Davoudi, Kourosh"],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025-08-01","date_published":"2025-08-01","updated_at":"2026-07-24T05:35:18Z","subjects":[],"languages":["en"],"rights":[],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/10155/1982","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Davoudi, Kourosh"]},{"key":"dc:creator","label":"Author","values":["Bakos, Steve"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2025-09-18T15:32:34Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2025-09-18T15:32:34Z"]},{"key":"dc:date.issued","label":"Date","values":["2025-08-01"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master of Science (MSc)"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Ontario Institute of Technology"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["en"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/10155/1982"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Knowledge Distillation (KD) enables the creation of compact student models that can retain much of the predictive power of larger teacher models. While these student models are invaluable for deployment in resource-constrained environments, the process of identifying an optimal student—balancing architecture and training hyperparameters—is often hindered by the extensive and computationally intensive search required. This thesis introduces the KD-Policy-Learning (KD-PL) framework, a novel approach designed to mitigate this challenge. KD-PL integrates an explicit caching mechanism for previously evaluated configurations and an adaptive proximity analysis module within a Reinforcement Learning (RL) agent. This agent performs joint Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) specifically for KD. By systematically avoiding redundant evaluations of identical configurations and accurately estimating the performance of similar, novel ones, our framework markedly improves search efficiency. Experimental results demonstrate that KD-PL significantly reduces the computational cost of discovering effective student models—achieving substantial wall-clock time savings (e.g., 70–93%) compared to strong baselines. This enhanced efficiency is particularly beneficial when working with large teacher models or under strict resource budgets, all while maintaining competitive student model accuracy."]},{"key":"dc:title","label":"Title","values":["Towards searching for the best student in a Knowledge Distillation framework"]}]}],"canonical_facts":{"dc:contributor.advisor":["Davoudi, Kourosh"],"dc:creator":["Bakos, Steve"],"dc:date.accessioned":["2025-09-18T15:32:34Z"],"dc:date.available":["2025-09-18T15:32:34Z"],"dc:date.issued":["2025-08-01"],"dc:description.abstract":["Knowledge Distillation (KD) enables the creation of compact student models that can retain much of the predictive power of larger teacher models. While these student models are invaluable for deployment in resource-constrained environments, the process of identifying an optimal student—balancing architecture and training hyperparameters—is often hindered by the extensive and computationally intensive search required. This thesis introduces the KD-Policy-Learning (KD-PL) framework, a novel approach designed to mitigate this challenge. KD-PL integrates an explicit caching mechanism for previously evaluated configurations and an adaptive proximity analysis module within a Reinforcement Learning (RL) agent. This agent performs joint Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) specifically for KD. By systematically avoiding redundant evaluations of identical configurations and accurately estimating the performance of similar, novel ones, our framework markedly improves search efficiency. Experimental results demonstrate that KD-PL significantly reduces the computational cost of discovering effective student models—achieving substantial wall-clock time savings (e.g., 70–93%) compared to strong baselines. This enhanced efficiency is particularly beneficial when working with large teacher models or under strict resource budgets, all while maintaining competitive student model accuracy."],"dc:identifier.uri":["https://hdl.handle.net/10155/1982"],"dc:language.iso":["en"],"dc:title":["Towards searching for the best student in a Knowledge Distillation framework"],"dc:type":["Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_name":["Master of Science (MSc)"],"thesis:institution_name":["University of Ontario Institute of Technology"]},"updated_at":"2026-07-24T05:35:18Z"}