{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/125607"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/125607","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Optimization and automation for efficient neural architecture design","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-02-04 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-02-04 without embargo terms","abstract_has_math":false,"creators":["Li, Yuhong"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Electrical & Computer Engr","degree_department":null,"school":null,"contributors":["Chen, Deming","Kim, Nam Sung","Wang, Yuxiong","Xiong, Jinjun","Dey, Debadeepta","Hao, Cong"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024-07-11","date_published":"2024-07-11","updated_at":"2026-07-22T22:25:02Z","subjects":["Deep Learning","Neural Networks","Neural Architecture Search","Computer Vision","Natural Language Processing"],"languages":["en","eng"],"rights":["Copyright 2024 Yuhong Li"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/125607","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Chen, Deming","Kim, Nam Sung","Wang, Yuxiong","Xiong, Jinjun","Dey, Debadeepta","Hao, Cong"]},{"key":"dc:creator","label":"Author","values":["Li, Yuhong"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2024-07-11","2024-08"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Electrical & Computer Engr"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Deep Learning","Neural Networks","Neural Architecture Search","Computer Vision","Natural Language Processing"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2024 Yuhong Li"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/125607"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-02-04 without embargo terms","The student, Yuhong Li, accepted the attached license on 2024-07-10 at 13:54.","The student, Yuhong Li, submitted this Dissertation for approval on 2024-07-10 at 14:04.","This Dissertation was approved for publication on 2024-07-11 at 08:43.","DSpace SAF Submission Ingestion Package generated from Vireo submission #21047 on 2025-02-04 at 21:04:54","As AI progresses, deep neural networks (DNNs) have become increasingly complex and resource-intensive, challenging their deployment on both cloud and edge systems. This dissertation addresses the critical challenges in neural architecture design by exploring various efficient search and optimization methods. It aims to deepen our understanding of the factors influencing DNN quality and develop automated processes to create optimized architectures for various applications, especially in resource-constrained settings. Structured into five chapters, this dissertation aims to address three central inquiries: • Efficient exploration within a defined design space: How do we navigate a predefined design space to not only explore efficiently but also identify and forge superior architectural designs? • Innovation beyond conventional boundaries: Machine-led design, guided by predefined algorithms and data, often lacks the creative novelty of researchers who utilize abstract thinking and diverse experiences. How can we not only break traditional confines but also refine and elevate innovative designs using automation? • Harmonizing hardware efficiency with quality: How do we strike a balance between the efficiency demands of the target hardware and the quality requirements of the model? Our journey starts with EDD, where we introduce a hardware-aware differentiable neural architecture search. This novel method employs a differentiable formulation that integrates both model and hardware variables into a single search space, allowing for the improvement of efficiency. The key innovations include enhanced predictions on massive classification datasets and better efficiency in energy and computation speed on the target hardware, leading to significant advancements in the quality of neural architectures searched. However, we notice that the latency of neural networks is easier to predict for well-defined search spaces compared to the quality of neural architectures. Also, search spaces are not always representable in a continuous form, such as varying downsampling rates across different convolutional layers in a CNN, which introduces discrete decision points. To address this challenge, we introduce GenNAS, a search algorithm that utilizes synthetic regression- based few-shot learning tasks to obtain estimated scores of networks that are highly correlated with their quality on large datasets. Furthermore, we observe that although GenNAS effectively navigates discrete search spaces and predicts accurately, evaluating each network individually remains time-consuming. We propose Eproxy, a sophisticated proxy- based approach that leverages self-supervised learning and few-shot techniques. By creating a more challenging synthetic proxy task, Eproxy enables faster evaluation and drastically reduces computational expenses. Moreover, we introduce a task search space that allows Eproxy to adapt to unconventional search spaces across various tasks. Although the proposed NAS algorithms can perform accurate predictions within enormous search spaces, these spaces usually consist of conventional components, such as standard convolutional layers, which limit the potential for innovative breakthroughs. Based on our observations that current models fail to efficiently handle long-range dependencies, we have designed a novel architectural component called Structured Global Convolution (SGConv). This component employs a multi-scale sub-kernel strategy, which significantly enhances the capture of long-range dependencies. We further explore the search space of the sparsity of SGConv and a mixed architecture with both SGConv and the attention as the components. By understanding the benefit of collaboration between human innovation and machine automation, we use the paradigms to explore the design of large language models (LLMs) using automation. Noticing that LLMs are primarily memory-bound due to the sequential nature of autoregressive decoding—where each token generated requires transferring large model parameters from memory, resulting in inefficient use of computational resources—we introduce Medusa. Medusa optimizes decoding efficiency by employing an adaptor that integrates additional decoding heads capable of predicting multiple subsequent tokens simultaneously, significantly enhancing throughput. We develop an automated framework that delivers the adapter from training to deployment. We further explore the search space such as the sparsity of the tree and the number of decoding heads. Moreover, we propose a simple hardware-aware profiling strategy for Medusa to predict its performance on the target devices with the consideration of batch sizes, sequence lengths, and sizes of LLMs. In conclusion, this dissertation addresses neural architecture design challenges through innovative optimization and automation methods, advancing AI. By exploring differentiable searches, synthetic evaluations, and innovations like SGConv and the Medusa framework, it enriches our understanding of neural network efficiency and sets innovative standards for AI applications. Each chapter contributes to deep learning design, enhancing applications from the mobile to the cloud and promoting sustainable AI evolution."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Optimization and automation for efficient neural architecture design"]}]}],"canonical_facts":{"dc:contributor":["Chen, Deming","Kim, Nam Sung","Wang, Yuxiong","Xiong, Jinjun","Dey, Debadeepta","Hao, Cong"],"dc:creator":["Li, Yuhong"],"dc:date":["2024-07-11","2024-08"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-02-04 without embargo terms","The student, Yuhong Li, accepted the attached license on 2024-07-10 at 13:54.","The student, Yuhong Li, submitted this Dissertation for approval on 2024-07-10 at 14:04.","This Dissertation was approved for publication on 2024-07-11 at 08:43.","DSpace SAF Submission Ingestion Package generated from Vireo submission #21047 on 2025-02-04 at 21:04:54","As AI progresses, deep neural networks (DNNs) have become increasingly complex and resource-intensive, challenging their deployment on both cloud and edge systems. This dissertation addresses the critical challenges in neural architecture design by exploring various efficient search and optimization methods. It aims to deepen our understanding of the factors influencing DNN quality and develop automated processes to create optimized architectures for various applications, especially in resource-constrained settings. Structured into five chapters, this dissertation aims to address three central inquiries: • Efficient exploration within a defined design space: How do we navigate a predefined design space to not only explore efficiently but also identify and forge superior architectural designs? • Innovation beyond conventional boundaries: Machine-led design, guided by predefined algorithms and data, often lacks the creative novelty of researchers who utilize abstract thinking and diverse experiences. How can we not only break traditional confines but also refine and elevate innovative designs using automation? • Harmonizing hardware efficiency with quality: How do we strike a balance between the efficiency demands of the target hardware and the quality requirements of the model? Our journey starts with EDD, where we introduce a hardware-aware differentiable neural architecture search. This novel method employs a differentiable formulation that integrates both model and hardware variables into a single search space, allowing for the improvement of efficiency. The key innovations include enhanced predictions on massive classification datasets and better efficiency in energy and computation speed on the target hardware, leading to significant advancements in the quality of neural architectures searched. However, we notice that the latency of neural networks is easier to predict for well-defined search spaces compared to the quality of neural architectures. Also, search spaces are not always representable in a continuous form, such as varying downsampling rates across different convolutional layers in a CNN, which introduces discrete decision points. To address this challenge, we introduce GenNAS, a search algorithm that utilizes synthetic regression- based few-shot learning tasks to obtain estimated scores of networks that are highly correlated with their quality on large datasets. Furthermore, we observe that although GenNAS effectively navigates discrete search spaces and predicts accurately, evaluating each network individually remains time-consuming. We propose Eproxy, a sophisticated proxy- based approach that leverages self-supervised learning and few-shot techniques. By creating a more challenging synthetic proxy task, Eproxy enables faster evaluation and drastically reduces computational expenses. Moreover, we introduce a task search space that allows Eproxy to adapt to unconventional search spaces across various tasks. Although the proposed NAS algorithms can perform accurate predictions within enormous search spaces, these spaces usually consist of conventional components, such as standard convolutional layers, which limit the potential for innovative breakthroughs. Based on our observations that current models fail to efficiently handle long-range dependencies, we have designed a novel architectural component called Structured Global Convolution (SGConv). This component employs a multi-scale sub-kernel strategy, which significantly enhances the capture of long-range dependencies. We further explore the search space of the sparsity of SGConv and a mixed architecture with both SGConv and the attention as the components. By understanding the benefit of collaboration between human innovation and machine automation, we use the paradigms to explore the design of large language models (LLMs) using automation. Noticing that LLMs are primarily memory-bound due to the sequential nature of autoregressive decoding—where each token generated requires transferring large model parameters from memory, resulting in inefficient use of computational resources—we introduce Medusa. Medusa optimizes decoding efficiency by employing an adaptor that integrates additional decoding heads capable of predicting multiple subsequent tokens simultaneously, significantly enhancing throughput. We develop an automated framework that delivers the adapter from training to deployment. We further explore the search space such as the sparsity of the tree and the number of decoding heads. Moreover, we propose a simple hardware-aware profiling strategy for Medusa to predict its performance on the target devices with the consideration of batch sizes, sequence lengths, and sizes of LLMs. In conclusion, this dissertation addresses neural architecture design challenges through innovative optimization and automation methods, advancing AI. By exploring differentiable searches, synthetic evaluations, and innovations like SGConv and the Medusa framework, it enriches our understanding of neural network efficiency and sets innovative standards for AI applications. Each chapter contributes to deep learning design, enhancing applications from the mobile to the cloud and promoting sustainable AI evolution."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/125607"],"dc:language":["en","eng"],"dc:rights":["Copyright 2024 Yuhong Li"],"dc:subject":["Deep Learning","Neural Networks","Neural Architecture Search","Computer Vision","Natural Language Processing"],"dc:title":["Optimization and automation for efficient neural architecture design"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Electrical & Computer Engr"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:02Z"}