{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/121553"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/121553","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"A good teacher is all you need","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-12-04 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-12-04 without embargo terms","abstract_has_math":false,"creators":["Farhat-Sabet, Sean"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Chen, Deming"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2023,"date_issued":"2023-08","date_published":"2023-08","updated_at":"2026-07-22T22:25:00Z","subjects":["Synthetic Data","Computer Science","Artificial Intelligence","Machine Learning","Pre-training","Finetuning","Knowledge Distillation","Contrastive Learning","Small Models","Training Efficiency","Generative Models","Diffusion Models"],"languages":["en","eng"],"rights":["Copyright 2023 Sean Farhat-Sabet"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/121553","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Chen, Deming"]},{"key":"dc:creator","label":"Author","values":["Farhat-Sabet, Sean"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2023-08","2023-07-20"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Synthetic Data","Computer Science","Artificial Intelligence","Machine Learning","Pre-training","Finetuning","Knowledge Distillation","Contrastive Learning","Small Models","Training Efficiency","Generative Models","Diffusion Models"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2023 Sean Farhat-Sabet"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/121553"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-12-04 without embargo terms","The student, Sean Farhat-Sabet, accepted the attached license on 2023-07-19 at 12:50.","The student, Sean Farhat-Sabet, submitted this Thesis for approval on 2023-07-19 at 14:14.","This Thesis was approved for publication on 2023-07-20 at 10:04.","DSpace SAF Submission Ingestion Package generated from Vireo submission #19743 on 2023-12-04 at 17:03:27","In this thesis, we tackle the problem of improving the performance of small machine learning models. A classic approach to enhance any model is to first pre-train it on an enormous, diverse dataset such as ImageNet, thereby equipping it with a strong feature backbone. Subsequently, finetuning this model on a desired downstream task results in generally better performance, compared to if it was only trained on the task. Larger models tend to benefit more from this pre-training paradigm, and several attempts have been made to minimize the drop in performance when the model size is decreased. We propose a simple, straightforward, and alternate approach that completely avoids the costly pre-training by (1) taking a large, publicly available pre-trained model, (2) finetuning it on the desired task, and (3) teaching its knowledge of that task to a target small model. Surprisingly, this leads to performance competitive with the pre-training paradigm, sometimes even surpassing it, while only using a fraction of the resources. Our approach can be viewed as designing a stronger knowledge distillation (KD) setup by explicitly considering the teacher’s knowledge and the student’s questions (transfer dataset), as well as introducing 2 variants of a novel knowledge transfer algorithm. The first is derived from our perspective of KD as a form of Noise Contrastive Estimation (NCE), thereby allowing any tool from contrastive learning to be used. We choose one, the Alignment/Uniformity metric, as an illustration. The second uses ideas from work on metrics for high-dimensional representations, specifically GULP. Lastly, we can gain a further boost in performance by augmenting the transfer dataset with synthetically generated samples from a publicly available, pre-trained, text-to-image generative diffusion model. We test our method on 2 small models across 5 visual recognition tasks, most of which are data-limited. When compared to when they were pre-trained and finetuned, our small models either surpass their performance or lag behind by at most 1.5%, while cutting training time up to 95%. Thus, we refer to our paradigm as: Don’t Pre-train, Teach (DPT)."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["A good teacher is all you need"]}]}],"canonical_facts":{"dc:contributor":["Chen, Deming"],"dc:creator":["Farhat-Sabet, Sean"],"dc:date":["2023-08","2023-07-20"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-12-04 without embargo terms","The student, Sean Farhat-Sabet, accepted the attached license on 2023-07-19 at 12:50.","The student, Sean Farhat-Sabet, submitted this Thesis for approval on 2023-07-19 at 14:14.","This Thesis was approved for publication on 2023-07-20 at 10:04.","DSpace SAF Submission Ingestion Package generated from Vireo submission #19743 on 2023-12-04 at 17:03:27","In this thesis, we tackle the problem of improving the performance of small machine learning models. A classic approach to enhance any model is to first pre-train it on an enormous, diverse dataset such as ImageNet, thereby equipping it with a strong feature backbone. Subsequently, finetuning this model on a desired downstream task results in generally better performance, compared to if it was only trained on the task. Larger models tend to benefit more from this pre-training paradigm, and several attempts have been made to minimize the drop in performance when the model size is decreased. We propose a simple, straightforward, and alternate approach that completely avoids the costly pre-training by (1) taking a large, publicly available pre-trained model, (2) finetuning it on the desired task, and (3) teaching its knowledge of that task to a target small model. Surprisingly, this leads to performance competitive with the pre-training paradigm, sometimes even surpassing it, while only using a fraction of the resources. Our approach can be viewed as designing a stronger knowledge distillation (KD) setup by explicitly considering the teacher’s knowledge and the student’s questions (transfer dataset), as well as introducing 2 variants of a novel knowledge transfer algorithm. The first is derived from our perspective of KD as a form of Noise Contrastive Estimation (NCE), thereby allowing any tool from contrastive learning to be used. We choose one, the Alignment/Uniformity metric, as an illustration. The second uses ideas from work on metrics for high-dimensional representations, specifically GULP. Lastly, we can gain a further boost in performance by augmenting the transfer dataset with synthetically generated samples from a publicly available, pre-trained, text-to-image generative diffusion model. We test our method on 2 small models across 5 visual recognition tasks, most of which are data-limited. When compared to when they were pre-trained and finetuned, our small models either surpass their performance or lag behind by at most 1.5%, while cutting training time up to 95%. Thus, we refer to our paradigm as: Don’t Pre-train, Teach (DPT)."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/121553"],"dc:language":["en","eng"],"dc:rights":["Copyright 2023 Sean Farhat-Sabet"],"dc:subject":["Synthetic Data","Computer Science","Artificial Intelligence","Machine Learning","Pre-training","Finetuning","Knowledge Distillation","Contrastive Learning","Small Models","Training Efficiency","Generative Models","Diffusion Models"],"dc:title":["A good teacher is all you need"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:00Z"}