Towards searching for the best student in a Knowledge Distillation framework

Knowledge Distillation (KD) enables the creation of compact student models that can retain much of the predictive power of larger teacher models. While these student models are invaluable for deployment in resource-constrained environments, the process of identifying an optimal student—balancing architecture and training hyperparameters—is often hindered by the extensive and computationally intensive search required. This thesis introduces the KD-Policy-Learning (KD-PL) framework, a novel approach designed to mitigate this challenge. KD-PL integrates an explicit caching mechanism for previously evaluated configurations and an adaptive proximity analysis module within a Reinforcement Learning (RL) agent. This agent performs joint Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) specifically for KD. By systematically avoiding redundant evaluations of identical configurations and accurately estimating the performance of similar, novel ones, our framework markedly improves search efficiency. Experimental results demonstrate that KD-PL significantly reduces the computational cost of discovering effective student models—achieving substantial wall-clock time savings (e.g., 70–93%) compared to strong baselines. This enhanced efficiency is particularly beneficial when working with large teacher models or under strict resource budgets, all while maintaining competitive student model accuracy.

Name thesis:degree_name

Master of Science (MSc)

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Ontario Institute of Technology

Year dc:date.issued

2025

Author dc:creator

• Bakos, Steve

Advisor dc:contributor.advisor

• Davoudi, Kourosh

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/10155/1982

OAI identifier oai:identifier

oai:ontariotechu.scholaris.ca:10155/1982