Function-Space Bayesian Learning: from Gaussian Processes to Bayesian Deep Learning

Bayesian methods provide a general and principled framework to quantify and update beliefs based on prior knowledge and observed evidence. This thesis presents my works about function-space Bayesian learning, whose priors and posteriors are specified and computed over the function-space. This thesis first focuses on Gaussian processes (GPs), a family of traditional function-space Bayesian models. To learn a suitable kernel for the GP prior, I propose a neural network structure that computes expressive kernel compositions. Because this structure is fully differentiable, we can learn the prior kernel from data with end-to-end gradient-based optimization. Moreover, I propose a new variational GP approximation to address computational bottlenecks in variational Gaussian processes. This variational GP achieves substantial improvement in computational complexity and approximates the true posterior with high fidelity. In the next part, I propose a general framework for function-space Bayesian learning that goes beyond GPs. I apply this framework in Bayesian deep learning and demonstrate its ability to train Bayesian neural networks with interpretable function-space priors. Furthermore, by exploiting a function-space perspective, I show that the hidden units in neural networks can be viewed as inter-domain inducing points so that I can draw an equivalence between finite neural networks and sparse Gaussian processes. In the last part, I employ Bayesian models for online memory selection and improve the robustness against imbalanced data streams in continual learning. Finally, I summarize the thesis and discuss potential future directions.

Department dc:contributor.department

Computer Science

Year dc:date.issued

2022

Author dc:creator

• Sun, Shengyang

Advisor dc:contributor.advisor

• Grosse, Roger

• Bayesian Deep Learning
• Bayesian Methods
• Gaussian Process
• Machine Learning
• Uncertainty Estimation

Statement dc:rights

• Attribution 4.0 International

Licence dc:rights.uri

• http://creativecommons.org/licenses/by/4.0/

Handle dc:identifier.uri

http://hdl.handle.net/1807/125369

OAI identifier oai:identifier

oai:utoronto.scholaris.ca:1807/125369