Advances in Hierarchical Probabilistic Multimodal Data Fusion

Multimodal data fusion is the process of integrating disparate data sources into a shared representation suitable for complex reasoning. As a result, one can make more precise inferences about the underlying phenomenon than is possible with each data source used in isolation. In the thesis we adopt a Bayesian view of multimodal data fusion, which formulates reasoning as posterior inference over latent variables. Within the Bayesian setting we present a novel method for data integration that we call lightweight data fusion (LDF). LDF addresses the case where the forward model for a subset of the data sources is unknown or poorly characterized. LDF leverages the remaining data sources to learn an inverse model suitable for posterior inference that combines both types of data. Additionally, we develop a multimodal extension to hierarchical Dirichlet processes (mmHDPs) where, in contrast to the setting for LDF, we lack observation-level correspondences across modalities and the data arise from an implicit latent variable model. Finally, we develop a novel representation for Dirichlet process and HDP mixture models that enables parallelization during inference and extends to more complex models including mmHDPs.

Name thesis:degree_name

Doctoral

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2022

Author dc:creator

• Dean, Christopher L.

Advisor dc:contributor.advisor

• Fisher III, John W.

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright MIT

Licence dc:rights.uri

• http://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/144802

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/144802