{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/120214"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/120214","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Robust and efficient neural networks: Algorithms, architectures, and circuits","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","abstract_has_math":false,"creators":["Dbouk, Hassan"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Electrical & Computer Engr","degree_department":null,"school":null,"contributors":["Shanbhag, Naresh","Schwing, Alexander","Li, Bo","Mehendale, Mahesh"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2023,"date_issued":"2023-05","date_published":"2023-05","updated_at":"2026-07-22T22:24:57Z","subjects":["Deep Learning","Efficient Inference","Adversarial Robustness","Quantization","Hardware Acceleration","Randomized Ensembles","Keyword Spotting"],"languages":["en","eng"],"rights":["Copyright 2023 Hassan Dbouk"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/120214","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Shanbhag, Naresh","Schwing, Alexander","Li, Bo","Mehendale, Mahesh"]},{"key":"dc:creator","label":"Author","values":["Dbouk, Hassan"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2023-05","2023-03-16"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Electrical & Computer Engr"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Deep Learning","Efficient Inference","Adversarial Robustness","Quantization","Hardware Acceleration","Randomized Ensembles","Keyword Spotting"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2023 Hassan Dbouk"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/120214"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","The student, Hassan Dbouk, accepted the attached license on 2023-03-15 at 11:02.","The student, Hassan Dbouk, submitted this Dissertation for approval on 2023-03-15 at 11:11.","This Dissertation was approved for publication on 2023-03-16 at 11:10.","DSpace SAF Submission Ingestion Package generated from Vireo submission #18874 on 2023-09-01 at 17:07:31","The recent success of deep learning has had a massive impact on our lives. The release of AlexNet, the winner of the Large Scale Visual Recognition Challenge in 2012, revitalized deep learning research. As a result, deep neural networks have achieved state-of-the-art results in various fields, often surpassing human-level accuracy and even beating world champions at their own strategy games. Today, most of the 'intelligence' achieved by deep nets is deployed on the cloud, where compute resources are abundant. However, there is an ever-growing interest in bringing this intelligence to the resource-constrained Edge, a requirement for many applications such as autonomous driving for instance. Deploying deep nets at the Edge imposes strict limitations on their complexity, which translates to sub-par performance. Furthermore, it has been observed that deep nets are inherently vulnerable to adversarial perturbations. Well-crafted imperceptible perturbations can fool undefended networks with deterministic success. These two challenges have been addressed mostly in isolation, with very few works proposing techniques for efficient and robust deep nets. Our research aims at bridging the gap between accuracy, robustness, and complexity, via a cross-layered approach. First, we tackle the accuracy vs. complexity trade-off for keyword spotting systems (KWS), by adopting an algorithm-hardware co-design approach. To the best our knowledge, this is the first work to propose using a Recurrent Attention Model (RAM), previously proposed for image classification, for KWS (KeyRAM algorithm) and the first IC implementation of RAM for KWS (KeyRAM IC). The proposed system adopts an in-memory computing (IMC)-based architecture and is taped-out in a 65nm CMOS process. The KeyRAM IC demonstrates up to 24× savings in the energy-delay-product over existing KWS implementations. Second, we identify the problem of aggressively quantizing lightweight deep nets. Traditional quantization techniques have been demonstrated on over-parameterized networks and often fail to aggressively quantize compact networks such as MobileNets. To that end, we propose DBQ, an efficient and differentiable multiple ternary branch quantizer for aggressively quantizing lightweight networks. DBQ successfully ternarizes lightweight networks, with minimal degradation in accuracy on ImageNet. Third, we propose generalized depthwise-separable (GDWS) convolutions to improve the robustness vs. complexity trade-off in deep nets. We derive efficient and optimal approximation algorithms for approximating pre-trained standard convolutions with GDWS ones. Post-training application of GDWS on adversarially-trained convolutional neural nets results in massive improvements in throughput, measured in frames-per-second, when mapped onto an NVIDIA Jetson board, while preserving robustness. We demonstrate the effectiveness of GDWS via extensive benchmarking across a variety of network architectures and datasets. Finally, we study the adversarial robustness of randomized ensemble classifiers (RECs), where one classifier is selected at random during inference. We establish theoretically that commonly employed robustness evaluation methods such as adaptive PGD provide a false sense of security in this setting. Subsequently, we propose a theoretically-sound and efficient adversarial attack algorithm (ARC) capable of compromising existing randomized ensemble defenses. Finally, we derive fundamental results regarding the theoretical limits of RECs, necessary and sufficient conditions for them to be useful, and more. Leveraging this new understanding, we propose a new boosting algorithm (BARRE) for training robust RECs, and empirically demonstrate its effectiveness at defending against strong adversaries across various network architectures and datasets."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Robust and efficient neural networks: Algorithms, architectures, and circuits"]}]}],"canonical_facts":{"dc:contributor":["Shanbhag, Naresh","Schwing, Alexander","Li, Bo","Mehendale, Mahesh"],"dc:creator":["Dbouk, Hassan"],"dc:date":["2023-05","2023-03-16"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","The student, Hassan Dbouk, accepted the attached license on 2023-03-15 at 11:02.","The student, Hassan Dbouk, submitted this Dissertation for approval on 2023-03-15 at 11:11.","This Dissertation was approved for publication on 2023-03-16 at 11:10.","DSpace SAF Submission Ingestion Package generated from Vireo submission #18874 on 2023-09-01 at 17:07:31","The recent success of deep learning has had a massive impact on our lives. The release of AlexNet, the winner of the Large Scale Visual Recognition Challenge in 2012, revitalized deep learning research. As a result, deep neural networks have achieved state-of-the-art results in various fields, often surpassing human-level accuracy and even beating world champions at their own strategy games. Today, most of the 'intelligence' achieved by deep nets is deployed on the cloud, where compute resources are abundant. However, there is an ever-growing interest in bringing this intelligence to the resource-constrained Edge, a requirement for many applications such as autonomous driving for instance. Deploying deep nets at the Edge imposes strict limitations on their complexity, which translates to sub-par performance. Furthermore, it has been observed that deep nets are inherently vulnerable to adversarial perturbations. Well-crafted imperceptible perturbations can fool undefended networks with deterministic success. These two challenges have been addressed mostly in isolation, with very few works proposing techniques for efficient and robust deep nets. Our research aims at bridging the gap between accuracy, robustness, and complexity, via a cross-layered approach. First, we tackle the accuracy vs. complexity trade-off for keyword spotting systems (KWS), by adopting an algorithm-hardware co-design approach. To the best our knowledge, this is the first work to propose using a Recurrent Attention Model (RAM), previously proposed for image classification, for KWS (KeyRAM algorithm) and the first IC implementation of RAM for KWS (KeyRAM IC). The proposed system adopts an in-memory computing (IMC)-based architecture and is taped-out in a 65nm CMOS process. The KeyRAM IC demonstrates up to 24× savings in the energy-delay-product over existing KWS implementations. Second, we identify the problem of aggressively quantizing lightweight deep nets. Traditional quantization techniques have been demonstrated on over-parameterized networks and often fail to aggressively quantize compact networks such as MobileNets. To that end, we propose DBQ, an efficient and differentiable multiple ternary branch quantizer for aggressively quantizing lightweight networks. DBQ successfully ternarizes lightweight networks, with minimal degradation in accuracy on ImageNet. Third, we propose generalized depthwise-separable (GDWS) convolutions to improve the robustness vs. complexity trade-off in deep nets. We derive efficient and optimal approximation algorithms for approximating pre-trained standard convolutions with GDWS ones. Post-training application of GDWS on adversarially-trained convolutional neural nets results in massive improvements in throughput, measured in frames-per-second, when mapped onto an NVIDIA Jetson board, while preserving robustness. We demonstrate the effectiveness of GDWS via extensive benchmarking across a variety of network architectures and datasets. Finally, we study the adversarial robustness of randomized ensemble classifiers (RECs), where one classifier is selected at random during inference. We establish theoretically that commonly employed robustness evaluation methods such as adaptive PGD provide a false sense of security in this setting. Subsequently, we propose a theoretically-sound and efficient adversarial attack algorithm (ARC) capable of compromising existing randomized ensemble defenses. Finally, we derive fundamental results regarding the theoretical limits of RECs, necessary and sufficient conditions for them to be useful, and more. Leveraging this new understanding, we propose a new boosting algorithm (BARRE) for training robust RECs, and empirically demonstrate its effectiveness at defending against strong adversaries across various network architectures and datasets."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/120214"],"dc:language":["en","eng"],"dc:rights":["Copyright 2023 Hassan Dbouk"],"dc:subject":["Deep Learning","Efficient Inference","Adversarial Robustness","Quantization","Hardware Acceleration","Randomized Ensembles","Keyword Spotting"],"dc:title":["Robust and efficient neural networks: Algorithms, architectures, and circuits"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Electrical & Computer Engr"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:57Z"}