{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/120107"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/120107","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Efficient transformer-based panoptic segmentation via knowledge distillation","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","abstract_has_math":false,"creators":["Zhang, Wentao"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Gui, Liangyan"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2023,"date_issued":"2023-05","date_published":"2023-05","updated_at":"2026-07-22T22:24:56Z","subjects":["Computer Science","Machine Learning","Deep Learning","Computer Vision","Knowledge Distillation"],"languages":["en","eng"],"rights":["Copyright 2023 Wentao Zhang"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/120107","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Gui, Liangyan"]},{"key":"dc:creator","label":"Author","values":["Zhang, Wentao"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2023-05","2023-04-27"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computer Science","Machine Learning","Deep Learning","Computer Vision","Knowledge Distillation"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2023 Wentao Zhang"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/120107"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","The student, Wentao Zhang, accepted the attached license on 2023-04-25 at 15:20.","The student, Wentao Zhang, submitted this Thesis for approval on 2023-04-25 at 15:30.","This Thesis was approved for publication on 2023-04-27 at 08:50.","DSpace SAF Submission Ingestion Package generated from Vireo submission #19171 on 2023-09-01 at 16:55:33","Knowledge distillation has been applied to various models in different domains. However, knowledge distillation on panoptic segmentation has not been studied so far. In this work, we focus on the knowledge distillation on transformer-based model. More specifically, we perform thorough analysis on the Mask2Former model, which is one of the state-of-the-art models. We found that both backbone and segmentation head are bottleneck of the model performance. To build an efficient transformer-based panoptic segmentation model, one of the best practice is to direct initialize the student model with part of the teacher’s parameters. We first worked on layer parameter initialization and parameter group consistent parameter selection for initialization. We then explored different distillation matching schemes between layers and of teacher and student. Finally, we researched different distillation loss, including adaptive matching-based prediction loss, masked generative distillation-based image feature loss, standard attention distillation loss, and deformable attention distillation loss. With all distillation approaches mentioned above, we trained Mask2Former-S(hallow), Mask2Former- T(hin), and Mask2Former-ST. Our ResNet-50 based models outperformed previous strong baselines, including Panoptic Segformer, MaX-DeepLab, MaskFormer, DETR, Panoptic- DeepLab and Panoptic-FPN with far fewer parameters and GFlops on MS COCO dataset. Additionally, our ResNet-18 based model ourperformed ResNet-50 based Panoptic-DeepLab and Panoptic-FPN with only 29.3% of the parameters."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Efficient transformer-based panoptic segmentation via knowledge distillation"]}]}],"canonical_facts":{"dc:contributor":["Gui, Liangyan"],"dc:creator":["Zhang, Wentao"],"dc:date":["2023-05","2023-04-27"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","The student, Wentao Zhang, accepted the attached license on 2023-04-25 at 15:20.","The student, Wentao Zhang, submitted this Thesis for approval on 2023-04-25 at 15:30.","This Thesis was approved for publication on 2023-04-27 at 08:50.","DSpace SAF Submission Ingestion Package generated from Vireo submission #19171 on 2023-09-01 at 16:55:33","Knowledge distillation has been applied to various models in different domains. However, knowledge distillation on panoptic segmentation has not been studied so far. In this work, we focus on the knowledge distillation on transformer-based model. More specifically, we perform thorough analysis on the Mask2Former model, which is one of the state-of-the-art models. We found that both backbone and segmentation head are bottleneck of the model performance. To build an efficient transformer-based panoptic segmentation model, one of the best practice is to direct initialize the student model with part of the teacher’s parameters. We first worked on layer parameter initialization and parameter group consistent parameter selection for initialization. We then explored different distillation matching schemes between layers and of teacher and student. Finally, we researched different distillation loss, including adaptive matching-based prediction loss, masked generative distillation-based image feature loss, standard attention distillation loss, and deformable attention distillation loss. With all distillation approaches mentioned above, we trained Mask2Former-S(hallow), Mask2Former- T(hin), and Mask2Former-ST. Our ResNet-50 based models outperformed previous strong baselines, including Panoptic Segformer, MaX-DeepLab, MaskFormer, DETR, Panoptic- DeepLab and Panoptic-FPN with far fewer parameters and GFlops on MS COCO dataset. Additionally, our ResNet-18 based model ourperformed ResNet-50 based Panoptic-DeepLab and Panoptic-FPN with only 29.3% of the parameters."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/120107"],"dc:language":["en","eng"],"dc:rights":["Copyright 2023 Wentao Zhang"],"dc:subject":["Computer Science","Machine Learning","Deep Learning","Computer Vision","Knowledge Distillation"],"dc:title":["Efficient transformer-based panoptic segmentation via knowledge distillation"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:56Z"}