Synthesizing controversial sentences for testing the brain-predictivity of language models

Recent research has seen the rise of powerful neural-network language models that are sufficiently computationally precise and neurally plausible as to serve as a jumping-off base for our understanding of language processing in the brain. Because these models have been developed for optimizing a similar objective (word prediction), their brain predictions are often correlated, even though the models differ along several architectural and conceptual features, yielding a major challenge for testing which model features are most relevant for predicting language processing in the brain. Here, we address this challenge by synthesizing new sentence stimuli that maximally expose the disagreement between the predictions of a set of language models ('controversial stimuli'), which would not naturally occur in large language corpora . To do so, we develop a platform for systematizing this sentence synthesis process, providing a way to test different model-based hypotheses easily and efficiently. An initial exploration with this platform has begun to give us some intuition for how choosing from different pools of candidate words affect the kinds of sentences produced, and what kinds of changes tend to produce controversial sentences. For example, we show that the disagreement score, or the maximum amount of disagreement between models for a sentence, converges. This approach will eventually allow us to determine which models perform in the most human-like way and are most successful in predicting language processing in the brain, thus hopefully leading to insights on the mechanisms of human language understanding.

Name thesis:degree_name

Master

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2021

Author dc:creator

• Rakocevic, Lara I.

Advisor dc:contributor.advisor

• Evelina Fedorenko and Noga Zaslavsky.

• Electrical Engineering and Computer Science.

Statement dc:rights

• MIT theses may be protected by copyright. Please reuse MIT thesis content according to the MIT Libraries Permissions Policy, which is available through the URL provided.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/130713

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/130713