Certifiably trustworthy deep learning systems at scale

Great advances in deep learning (DL) have led to state-of-the-art performance on a wide range of challenging tasks. However, along with the rapid deployment of DL systems, several trustworthy threats arise, such as weak robustness against stealthy noise perturbations and natural transformations, bias across different subgroups, and lack of numerical reliability. These trustworthy threats have raised great concerns, especially when deploying DL systems in safety-critical scenarios such as autonomous driving and facial recognition for safeguarding. To defend against these common trustworthy threats, this thesis systematically proposes or enhances certification approaches and certified training approaches for DL systems, especially for large-scale DL systems. A certification approach can guarantee some properties of the DL system under some trustworthiness properties. For instance, the robustness certification approach can guarantee the worst-case test accuracy when the attacker imposes any input perturbations or transformations within some bounded range. A certified training approach can improve the DL system’s guaranteed trustworthiness under a certain property by training the DL model, e.g., improving the guaranteed test accuracy above. This thesis begins with a systematic taxonomy of certification and certified training approaches. Then for several critical trustworthiness properties, this thesis proposes the corresponding certification and certified training approaches that lead to state-of-the-art tightness and scalability. These approaches are motivated by a few core principles, including dual problem analysis for randomized smoothing, general cutting planes for bound propagation, stratified sampling, subpopulation decomposition, and abstract interpretation. The effectiveness of the proposed approaches is supported by both theoretical analyses and empirical evaluations. The thesis is concluded with a discussion of limitations, challenges, and future directions towards achieving fully certifiable, reliable, and scalable machine learning. In summary, this thesis enables certification of various trustworthy properties for DL systems up to millions of parameters, representing a major step in certified deep learning, an important research topic in machine learning, computer security, and software engineering.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Li, Linyi

Contributors dc:contributor

• Li, Bo
• Xie, Tao
• Gunter, Carl A.
• Kolter, J. Zico

• Deep Learning
• Certification
• Verification
• Trustworthy Machine Learning
• Machine Learning Security

Statement dc:rights

• Copyright 2023 Linyi Li

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/121953

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F121953