Robust and efficient neural networks: Algorithms, architectures, and circuits

The recent success of deep learning has had a massive impact on our lives. The release of AlexNet, the winner of the Large Scale Visual Recognition Challenge in 2012, revitalized deep learning research. As a result, deep neural networks have achieved state-of-the-art results in various fields, often surpassing human-level accuracy and even beating world champions at their own strategy games. Today, most of the 'intelligence' achieved by deep nets is deployed on the cloud, where compute resources are abundant. However, there is an ever-growing interest in bringing this intelligence to the resource-constrained Edge, a requirement for many applications such as autonomous driving for instance. Deploying deep nets at the Edge imposes strict limitations on their complexity, which translates to sub-par performance. Furthermore, it has been observed that deep nets are inherently vulnerable to adversarial perturbations. Well-crafted imperceptible perturbations can fool undefended networks with deterministic success. These two challenges have been addressed mostly in isolation, with very few works proposing techniques for efficient and robust deep nets. Our research aims at bridging the gap between accuracy, robustness, and complexity, via a cross-layered approach. First, we tackle the accuracy vs. complexity trade-off for keyword spotting systems (KWS), by adopting an algorithm-hardware co-design approach. To the best our knowledge, this is the first work to propose using a Recurrent Attention Model (RAM), previously proposed for image classification, for KWS (KeyRAM algorithm) and the first IC implementation of RAM for KWS (KeyRAM IC). The proposed system adopts an in-memory computing (IMC)-based architecture and is taped-out in a 65nm CMOS process. The KeyRAM IC demonstrates up to 24× savings in the energy-delay-product over existing KWS implementations. Second, we identify the problem of aggressively quantizing lightweight deep nets. Traditional quantization techniques have been demonstrated on over-parameterized networks and often fail to aggressively quantize compact networks such as MobileNets. To that end, we propose DBQ, an efficient and differentiable multiple ternary branch quantizer for aggressively quantizing lightweight networks. DBQ successfully ternarizes lightweight networks, with minimal degradation in accuracy on ImageNet. Third, we propose generalized depthwise-separable (GDWS) convolutions to improve the robustness vs. complexity trade-off in deep nets. We derive efficient and optimal approximation algorithms for approximating pre-trained standard convolutions with GDWS ones. Post-training application of GDWS on adversarially-trained convolutional neural nets results in massive improvements in throughput, measured in frames-per-second, when mapped onto an NVIDIA Jetson board, while preserving robustness. We demonstrate the effectiveness of GDWS via extensive benchmarking across a variety of network architectures and datasets. Finally, we study the adversarial robustness of randomized ensemble classifiers (RECs), where one classifier is selected at random during inference. We establish theoretically that commonly employed robustness evaluation methods such as adaptive PGD provide a false sense of security in this setting. Subsequently, we propose a theoretically-sound and efficient adversarial attack algorithm (ARC) capable of compromising existing randomized ensemble defenses. Finally, we derive fundamental results regarding the theoretical limits of RECs, necessary and sufficient conditions for them to be useful, and more. Leveraging this new understanding, we propose a new boosting algorithm (BARRE) for training robust RECs, and empirically demonstrate its effectiveness at defending against strong adversaries across various network architectures and datasets.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Dbouk, Hassan

Contributors dc:contributor

• Shanbhag, Naresh
• Schwing, Alexander
• Li, Bo
• Mehendale, Mahesh

• Deep Learning
• Efficient Inference
• Adversarial Robustness
• Quantization
• Hardware Acceleration
• Randomized Ensembles
• Keyword Spotting

Statement dc:rights

• Copyright 2023 Hassan Dbouk

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/120214

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F120214