{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/105133"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/105133","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Slimmable neural networks for edge devices","abstract":"While methods based on deep learning have witnessed major breakthroughs in machine perception and generative modeling, the problem of how to run neural networks within latency budget for edge devices remains unsolved. This thesis presents a new approach to train a single neural network executable at arbitrary widths for instant and adaptive accuracy-efficiency trade-offs at runtime. First a simple and general method is presented to train a single neural network executable at different widths (number of channels in a layer). The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve ImageNet classification accuracy similar to (and in many cases better than) that of individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. We visualize and discuss the learned features of slimmable networks. Further, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. In addition, we propose two improved training techniques for US-Nets, named the sandwich rule and the inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate a FLOPs-Accuracy spectrum of network architectures. Finally, we demonstrate an application to search for channel number configurations based on proposed slimmable networks.","abstract_html":"While methods based on deep learning have witnessed major breakthroughs in machine perception and generative modeling, the problem of how to run neural networks within latency budget for edge devices remains unsolved. This thesis presents a new approach to train a single neural network executable at arbitrary widths for instant and adaptive accuracy-efficiency trade-offs at runtime. First a simple and general method is presented to train a single neural network executable at different widths (number of channels in a layer). The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve ImageNet classification accuracy similar to (and in many cases better than) that of individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. We visualize and discuss the learned features of slimmable networks. Further, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. In addition, we propose two improved training techniques for US-Nets, named the sandwich rule and the inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate a FLOPs-Accuracy spectrum of network architectures. Finally, we demonstrate an application to search for channel number configurations based on proposed slimmable networks.","abstract_has_math":false,"creators":["Yu, Jiahui"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Electrical & Computer Engr","degree_department":null,"school":null,"contributors":["Huang, Thomas S."],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2019,"date_issued":"2019-08-23T20:44:31Z","date_published":"2019-08-23T20:44:31Z","updated_at":"2026-07-22T22:24:44Z","subjects":["slimmable networks"],"languages":["en"],"rights":["Copyright 2019 Jiahui Yu"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/105133","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Huang, Thomas S."]},{"key":"dc:creator","label":"Author","values":["Yu, Jiahui"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2019-08-23T20:44:31Z","2021-08-24T09:15:34Z","2019-02-15","2019-05"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Electrical & Computer Engr"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["slimmable networks"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2019 Jiahui Yu"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/105133"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["While methods based on deep learning have witnessed major breakthroughs in machine perception and generative modeling, the problem of how to run neural networks within latency budget for edge devices remains unsolved. This thesis presents a new approach to train a single neural network executable at arbitrary widths for instant and adaptive accuracy-efficiency trade-offs at runtime. First a simple and general method is presented to train a single neural network executable at different widths (number of channels in a layer). The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve ImageNet classification accuracy similar to (and in many cases better than) that of individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. We visualize and discuss the learned features of slimmable networks. Further, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. In addition, we propose two improved training techniques for US-Nets, named the sandwich rule and the inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate a FLOPs-Accuracy spectrum of network architectures. Finally, we demonstrate an application to search for channel number configurations based on proposed slimmable networks.","Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2021-05-01","The student, Jiahui Yu, accepted the attached license on 2019-02-14 at 14:34.","The student, Jiahui Yu, submitted this Thesis for approval on 2019-02-14 at 14:42.","This Thesis was approved for publication on 2019-02-15 at 11:18.","DSpace SAF Submission Ingestion Package generated from Vireo submission #13390 on 2019-08-22 at 16:19:49","Made available in DSpace on 2019-08-23T20:44:31Z (GMT). No. of bitstreams: 2 YU-THESIS-2019.pdf: 1268760 bytes, checksum: c091ef8a839188e9d52d208dee832b8a (MD5) LICENSE.txt: 4206 bytes, checksum: 1b6cf1c051b15c1073c51d0ad5e1abd0 (MD5) Previous issue date: 2019-02-15","Embargo set by: Seth Robbins for item 112252 Lift date: 2021-08-23T20:44:50Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 112252 Lift date: 2021-08-23T20:46:41Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 112252 Lift date: 2021-08-23T20:47:38Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 112252 Lift date: 2021-08-23T20:48:32Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Limited Restriction Lifted for Item 112252 on 2021-08-24T09:15:34Z."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Slimmable neural networks for edge devices"]}]}],"canonical_facts":{"dc:contributor":["Huang, Thomas S."],"dc:creator":["Yu, Jiahui"],"dc:date":["2019-08-23T20:44:31Z","2021-08-24T09:15:34Z","2019-02-15","2019-05"],"dc:description":["While methods based on deep learning have witnessed major breakthroughs in machine perception and generative modeling, the problem of how to run neural networks within latency budget for edge devices remains unsolved. This thesis presents a new approach to train a single neural network executable at arbitrary widths for instant and adaptive accuracy-efficiency trade-offs at runtime. First a simple and general method is presented to train a single neural network executable at different widths (number of channels in a layer). The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve ImageNet classification accuracy similar to (and in many cases better than) that of individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. We visualize and discuss the learned features of slimmable networks. Further, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. In addition, we propose two improved training techniques for US-Nets, named the sandwich rule and the inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate a FLOPs-Accuracy spectrum of network architectures. Finally, we demonstrate an application to search for channel number configurations based on proposed slimmable networks.","Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2021-05-01","The student, Jiahui Yu, accepted the attached license on 2019-02-14 at 14:34.","The student, Jiahui Yu, submitted this Thesis for approval on 2019-02-14 at 14:42.","This Thesis was approved for publication on 2019-02-15 at 11:18.","DSpace SAF Submission Ingestion Package generated from Vireo submission #13390 on 2019-08-22 at 16:19:49","Made available in DSpace on 2019-08-23T20:44:31Z (GMT). No. of bitstreams: 2 YU-THESIS-2019.pdf: 1268760 bytes, checksum: c091ef8a839188e9d52d208dee832b8a (MD5) LICENSE.txt: 4206 bytes, checksum: 1b6cf1c051b15c1073c51d0ad5e1abd0 (MD5) Previous issue date: 2019-02-15","Embargo set by: Seth Robbins for item 112252 Lift date: 2021-08-23T20:44:50Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 112252 Lift date: 2021-08-23T20:46:41Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 112252 Lift date: 2021-08-23T20:47:38Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 112252 Lift date: 2021-08-23T20:48:32Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Limited Restriction Lifted for Item 112252 on 2021-08-24T09:15:34Z."],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/105133"],"dc:language":["en"],"dc:rights":["Copyright 2019 Jiahui Yu"],"dc:subject":["slimmable networks"],"dc:title":["Slimmable neural networks for edge devices"],"dc:type":["text"],"thesis:degree_discipline":["Electrical & Computer Engr"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:44Z"}