Analyzing Multimodal Interactions through Improved Partial Information Decomposition Estimation

Multimodal AI aims to build comprehensive models by integrating information from diverse sensory inputs such as text, audio, and vision. However, significant challenges remain in understanding how these different modalities interact and contribute to downstream tasks. In particular, we seek to characterize how modalities complement each other, overlap in the information they convey, or contribute jointly to patterns that are not clear from any single modality alone. To address this, we propose novel methods for quantifying these multimodal interactions using information-theoretic techniques. Specifically, we will introduce a novel estimator for Partial Information Decomposition (PID) using normalizing flows, with the ability to scale well to high-dimensional data. We also develop a new framework for estimating pointwise PID, which provides insights into how individual data points contribute to information sharing and interactions across modalities, and show how to apply this framework for anomaly detection. We demonstrate the effectiveness of our methods on a variety of high-dimensional datasets, including both synthetic and real-world multimodal data such as videos.

Name thesis:degree_name

Master

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2025

Author dc:creator

• Balachandran, Adithya S.

Advisor dc:contributor.advisor

• Liang, Paul Pu

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright retained by author(s)

Licence dc:rights.uri

• https://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/162503

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/162503