Approximate Bayesian inference and optimal transport

In this thesis, we propose novel solutions to the problems of paramteric variational inference and generative modeling. One center problem in parameteric Bayesian inference is that high-dimensional integration for computing normalizing constant is intractable, since numerical error can accumulate to an intolerable level under hundreds of multiple integration, given main stream machine learning models can easily introduce hundreds of thousands of parameters. Among many exact or approximate methods trying to address this issue, variational inference contains a various family of powerful approximates. It is originated from particle physics, and has proved accuracy in many statistical problems combined with modern convex optimization techniques and computing facility. Yet its theoretical base still needs to be explored. In this work, we examine the frequentist consistency of mean-field variational inference, which include the effect from latent parameters. This complements the previous theory which only deals with global parameters. A main drawback of mean-field inference is underestimation of uncertainty, which undermines its range of application. In this work we propose variational weighted likelihood bootstrap, a method combining mean-field methods with Bayesian bootstrap. This makes it possible to draw accurate uncertainty characterization in the mean-field framework. We also connect the optimal transport(OT) theory with variational inference by addressing the problem from OT perspective. In general, distance measure in optimal transport theory, that is, Wasserstein distance, is a much weaker measure than those studied in Bayesian learning, such as total variation, Kullback-Leiber divergence and Jensen-Shannon divergence. This property is especially useful in the case where target distribution resides in a low-dimensional manifold. In this work we exploit the OT theory and try to combine this advantage with variational inference. This leads to the convex iterative structure we introduced. Generative models are a class of probabilistic models which focus more on directly generate observations rather than traditional machine learning tasks such as density estimation. This class has been found powerful in conducting unsupervised learning, for example, image generation, video prediction and 3d object generation. Recently, it has also been employed for posterior generation. However, to be best of our knowledge, there has not been work on combining this type of model directly with W2 theory, which performs better than generic generative models due to its geometric property. In this work, we use generative modeling techniques to find optimal solution that can generate independent samples effectively. In addition, the construction of our parametrization is dictated by previous work on Monge problems. In the discrete case, it approaches posterior faster than a log-likelihood based method, with Monte Carlo approximation and numerical tricks such as softmax. Moreover, we verify the effectiveness of this model carefully with theoretical justification. In particular, by showing a Berstein–von Mises type of convergence, we prove a normal posterior can be approximated effectively and by a minimal number of parameters.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Statistics

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2022

Author dc:creator

• Han, Wei

Contributors dc:contributor

• Yang, Yun
• Shao, Xiaofeng
• Liang, Feng
• Chen, Xiaohui

• Approximate Inference
• Bayesian Inference

Statement dc:rights

• Copyright 2022 Wei Han

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/116188

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F116188