Bayesian sparsity learning with variational automatic relevance determination

Automatic Relevance Determination (ARD) is a well-regarded Bayesian approach for feature selection, where each feature’s relevance is encoded in a hyper-parameter that is automatically tuned through the data. However, estimating the ARD prior via the evidence function poses significant computational challenges, with no closed-form solution and scalability issues. Existing ARD research primarily focuses on algorithm development, with limited theoretical understanding of its properties. In this thesis, we introduce Variational Automatic Relevance Determination (VARD), a novel approach that estimates the ARD prior efficiently through a variational method. We examine the statistical properties of VARD in the context of high-dimensional linear regression, providing convergence guarantees for both parameter estimation and variable selection. Additionally, we extend the VARD framework to additive models, enabling simultaneous estimation of smoothness and relevance for each feature. The first part of this thesis studies the ARD procedure within high-dimensional linear regression under sparsity assumptions. Our proposed VARD method approximates the posterior distribution with independent Gaussian distributions for each regression coefficient, where some distributions converge to a point mass at zero, automatically excluding irrelevant variables. We establish convergence results and present an efficient coordinate descent algorithm to implement VARD, demonstrating its empirical performance on simulated datasets. In the second part, we extend VARD to sparse additive models in high-dimensional settings. VARD uniquely enables independent smoothness estimation for each feature, distinguishing whether a feature’s effect on the response is zero, linear, or nonlinear. An efficient coordinate descent algorithm further supports this implementation. Empirical evaluations on simulated and real-world data highlight VARD’s advantages over alternative variable selection methods for additive models.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Statistics

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Liu, Zihe

Contributors dc:contributor

• Liu, Jingbo
• Yang, Yun
• Liang, Feng
• Chen, Yuguo

• Sparsity
• Automatic Relevance Determination
• Em- Pirical Bayes
• Variational Technique
• High-dimensional Linear Regression
• Generalized Additive Model
• Convergence

Statement dc:rights

• Copyright 2024 Zihe Liu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/127191

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F127191