Distribution-free uncertainty quantification for deep learning

The integration of sophisticated deep learning models into critical domains, such as healthcare, autonomous vehicles, and the legal system, is increasingly becoming a trend. These models offer significant potential for enhancing outcomes efficiently, but their adoption raises crucial challenges related to model confidence, uncertainty communication, and risk management. Uncertainty Quantification (UQ) is a key framework that addresses these issues by providing a systematic way to assess and act on the reliability of model predictions. This thesis focuses on distribution-free UQ methods, which require minimal assumptions about the data distribution and model specifics, making them highly applicable across various deep learning applications. We first investigate the problem of full calibration for deep learning classifiers, aiming to address the over-confidence or under-confidence typically observed in such classifiers. Then, we discuss methods to convert a model's output into actionable uncertainty information, expressed as prediction intervals and prediction sets. In particular, we focus on the construction of prediction intervals and sets with rigorous coverage or risk control guarantees, typically provided by conformal prediction tools. Finally, we explore the problem of UQ for natural language generation (NLG), for which we apply a graph-based approach using the similarity graph of multiple sampled responses. Through exploring model calibration, risk-controlling prediction sets, conformal prediction intervals, and UQ for NLG, this thesis aims to advance the understanding and implementation of UQ in high-stakes decision-making environments.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Lin, Zhen

Contributors dc:contributor

• Sun, Jimeng
• Rehg, James M
• Tong, Hanghang
• Romano, Yaniv

• Uncertainty Quantification
• Deep Learning
• Machine Learning
• Conformal Prediction
• Prediction Set
• Neural Networks
• Cross-sectional Time-series
• Healthcare Applications

Statement dc:rights

• Copyright 2024 Zhen Lin

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124231

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124231