Applications of diffusion processes: machine learning, optimization, and sampling

Many problems in machine learning and statistics today exhibit tremendous size in various parameters, to the point that they can often be conceived of as infinite. Doing so enables the use of mathematical tools for continuous systems in their analysis, which entails a careful consideration of the relationship between the discrete system in question and its continuous approximation. The work presented in this thesis uses this approach to study problems in optimization, sampling, and learning, by modeling the phenomena of interest with diffusion processes. In the first part, we bridge a short- and long-timescale understanding of optimization in non-convex settings using a noisy gradient-based method, the Langevin algorithm, a discretization of the eponymous diffusion process. In the second part, we use a control-theoretic framework to examine generative models specified by infinite noisy function compositions, corresponding to nonlinear diffusion processes, and demonstrate their expressiveness as well as methods for performing sampling and inference on them. Subsequently, we study the specific case where these models constitute the infinite-depth limit of feedforward neural networks, and discuss computational aspects of performing variational inference with them. Finally, we investigate probability distributions that can be viewed as the mean-field limit of the weights in very wide neural networks, and contrast control-theoretically optimal and Langevin dynamics vis-\`a-vis an entropically regularized risk minimization objective. We show why there is an exponential gap in the general case, and close by illustrating a setting where naive gradient-based dynamics in fact coincide exactly with those specified by optimal control: mirror Langevin dynamics, corresponding to the continuous limit of noisy mirror descent.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2022

Author dc:creator

• Tzen, Belinda

Contributors dc:contributor

• Raginsky, Maxim
• Jiang, Nan
• Rigollet, Philippe
• Srikant, Rayadurgam

• machine learning
• theory

Statement dc:rights

• Copyright 2022 Belinda Tzen

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/115878

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F115878