Priors in finite and infinite Bayesian convolutional neural networks
type: CreativeWork
description: Bayesian neural networks (BNNs) have undergone many changes since the seminal work of Neal [Nea96]. Advances in approximate inference and the use of GPUs have scaled BNNs to larger data sets, and much higher layer and parameter counts. Yet, the priors used for BNN parameters have remained essentially the same. The isotropic Gaussian prior introduced by Neal, where each element of the weights and biases is drawn independently from a Gaussian, is still used almost everywhere. This thesis seeks to undo the neglect in the development of priors for BNNs, especially convolutional BNNs, using a two-pronged approach. First, I theoretically examine the effect of the Gaussian isotropic prior on the distribution over functions of a deep BNN prior. I show that, as the number of channels of a convolutional BNN goes to infinity, its output converges in distribution to a Gaussian process (GP). Thus, we can draw rough conclusions about the function-space of finite BNNs by looking at the mean and covariance of their limiting GPs. The limiting GP itself performs surprisingly well at image classification, suggesting that knowledge encoded in the convolutional neural network (CNN) architecture, as opposed to the learned features, plays a larger role than previously thought. Examining the derived CNN kernel shows that, if the weights are independent, the output of the limiting GP loses translation equivariance. This is an important inductive bias for learning from images. We can prevent this loss by introducing spatial correlations in the weight prior of a Bayesian CNN, which still results in a GP in the infinite width limit. The second prong is an empirical methodology for identifying new priors for BNNs. Since BNNs are often considered to underfit, I examine the empirical distribution of weights learned using stochastic gradient descent (SGD). The resulting weight distributions tend to have heavier tails than a Gaussian, and display strong spatial correlations in CNNs. I incorporate t…
abstract: Bayesian neural networks (BNNs) have undergone many changes since the seminal work of Neal [Nea96]. Advances in approximate inference and the use of GPUs have scaled BNNs to larger data sets, and much higher layer and parameter counts. Yet, the priors used for BNN parameters have remained essentially the same. The isotropic Gaussian prior introduced by Neal, where each element of the weights and biases is drawn independently from a Gaussian, is still used almost everywhere. This thesis seeks to undo the neglect in the development of priors for BNNs, especially convolutional BNNs, using a two-pronged approach. First, I theoretically examine the effect of the Gaussian isotropic prior on the distribution over functions of a deep BNN prior. I show that, as the number of channels of a convolutional BNN goes to infinity, its output converges in distribution to a Gaussian process (GP). Thus, we can draw rough conclusions about the function-space of finite BNNs by looking at the mean and covariance of their limiting GPs. The limiting GP itself performs surprisingly well at image classification, suggesting that knowledge encoded in the convolutional neural network (CNN) architecture, as opposed to the learned features, plays a larger role than previously thought. Examining the derived CNN kernel shows that, if the weights are independent, the output of the limiting GP loses translation equivariance. This is an important inductive bias for learning from images. We can prevent this loss by introducing spatial correlations in the weight prior of a Bayesian CNN, which still results in a GP in the infinite width limit. The second prong is an empirical methodology for identifying new priors for BNNs. Since BNNs are often considered to underfit, I examine the empirical distribution of weights learned using stochastic gradient descent (SGD). The resulting weight distributions tend to have heavier tails than a Gaussian, and display strong spatial correlations in CNNs. I incorporate t…
author: Garriga Alonso, Adrià
language: eng
keywords: Bayesian neural networks, Machine learning, Gaussian processes, Infinitely wide neural networks, Cold-posterior effect, Convolutional neural networks, Bayesian statistics, Neural tangent kernel
license: https://creativecommons.org/licenses/by/4.0/
date published: 2021-12-01
date modified: 2026-07-22T22:24:16Z
provider: Cambridge University
publisher: Networked Digital Library of Theses and Dissertations
type: WebPage
source organization: University of Cambridge

PropertyValue
property ID: DOI
value: 10.17863/CAM.93557