Connecting Deep Learning Models to the Human Brain

In this thesis, we introduce innovative methodologies for connecting new deep learning models, particularly models that integrate vision and language with human brain processing. These models have shown remarkable advancements in tasks such as object recognition, scene classification, and language processing, achieving near-human accuracy in some cases. This raises intriguing questions about how closely the computations and geometric structure of these models mirror that of the human brain. Our method starts with measuring brain activity in response to vision and language stimuli and then exposes these stimuli to deep learning models to collect their internal activations. We analyze the similarity between these activations and brain activity using a specific representational distance metric. We focus on introducing statistical algorithms to assess whether one model is significantly more similar with the brain than another. Through our novel methodology, we assess whether there’s a more significant correlation between brain regions and multimodal models compared to unimodal ones. Our investigation reveals brain areas associated with vision-language integration and models of vision-language integration that are potentially most similar to the brain.

Name thesis:degree_name

Master

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2024

Author dc:creator

• Subramaniam, Vighnesh

Advisor dc:contributor.advisor

• Katz, Boris

Statement dc:rights

• Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
• Copyright retained by author(s)

Licence dc:rights.uri

• https://creativecommons.org/licenses/by-nc-nd/4.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/156797

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/156797