Scalable Gaussian Processes: Advances in Iterative Methods and Pathwise Conditioning

Driven by advances in hardware for massively-parallel computation, machine learning models trained on large amounts of data have become capable of accomplishing complex tasks, such as generating realistic images or maintaining conversations in natural language. However, the inability to know when they don't know often leads to overconfidence and hallucinations. Gaussian processes are a powerful framework for uncertainty-aware function approximation and sequential decision-making. Unfortunately, their classical formulation does not scale gracefully to large amounts of data and modern hardware for massively-parallel computation, prompting many researchers to develop techniques which improve their scalability. This dissertation focuses on the powerful combination of iterative methods and pathwise conditioning to develop methodological contributions which facilitate the use of Gaussian processes in modern large-scale settings. By combining these two techniques synergistically, expensive computations are expressed as solutions to systems of linear equations and obtained by leveraging iterative linear system solvers. This drastically reduces memory requirements, facilitating application to significantly larger amounts of data, and introduces matrix multiplication as the main computational operation, which is ideal for modern hardware. In particular, this dissertation introduces stochastic gradient algorithms as a computationally efficient method to solve linear systems iteratively. To this end, custom optimisation objectives, stochastic gradient estimators, and variance reduction techniques are developed and analysed. Empirically, the proposed methods achieve state-of-the-art performance on large- scale regression, Bayesian optimisation, and molecular binding affinity prediction tasks. Additionally, generic improvements, which are applicable to any iterative linear system solver in the context of Gaussian processes, are contributed, leading to computational speed-ups of up to 72x compared to established approaches. Furthermore, iterative methods and pathwise conditioning are combined with structured linear algebra techniques to attain even greater scalability, which is demonstrated on real-world datasets with up to five million examples, including robotics, automated machine learning, and climate modelling applications.

Name dc:type.qualificationname

Doctor of Philosophy (PhD)

Level dc:type.qualificationlevel

Doctoral

Grantor dc:publisher.institution

University of Cambridge

Year dc:date.issued

2025

Author dc:creator

• Lin, Jihao Andreas

Advisor dc:contributor.advisor

• Hernández-Lobato, José Miguel

• Gaussian process
• iterative methods
• linear system solver
• machine learning
• pathwise conditioning

Licence

• https://www.repository.cam.ac.uk/bitstreams/5dc1bc16-06b9-4d21-8b64-ffaa8b42156f/download
• http://purl.org/NET/rdflicense/allrightsreserved

DOI dc:identifier.doi

https://doi.org/10.17863/CAM.119535

OAI identifier oai:identifier

oai:www.repository.cam.ac.uk:1810/386221