Open-vocabulary brain-to-text decoding via cross-modal transfer with large language models

Recent research on transformer-based large language models and large-scale pretraining has shown unprecedented success in the field of natural language processing and beyond. A slew of previous work demonstrated that the rich knowledge embedded in large language models can be effective transferred across different modalities, to solve a wider range of discrinimative and generative tasks, e.g., vision-language understanding and robotic planning. In this thesis, we investigate an uncharted area of cross-modal transfer with large language models, decoding brain signals. State-of-the-art brain-to-text systems have achieved initial success in decoding language directly from brain signals using modern machine learning models such as recurrent neural networks. However, current approaches are limited to small closed vocabularies which are far from enough for natural communication. Additionally, most of the high-performing approaches require data from invasive devices (e.g., ECoG). To address the existing limitations, we extend the brain-to-text decoding problem to a more challenging open vocabulary setting, where the decoding is performed on the entire English vocabulary that is 100x larger than previous work. Furthermore, we decode from Electroencephalography (EEG) data which is non-invasive but more noisy. To handle the huge vocabulary size with a small amount of parallel EEG-text data, we propose a novel model leveraging pretrained large language models. The key idea is to learn a good brain-language alignment between the ``brain embedding space'' and the ``pretrained language embedding space'' so that we can directly leverage the power of a large language model, e.g., BART. Results show promising EEG-to-Text decoding performance with a BLEU-1 score of 40.1%. Moreover, in-depth analysis shows that the proposed model has strong scalability and robustness with respect to the data source, which is essential towards building a large-scale brain-language foundation model. To further demonstrate the effectiveness of the EEG-to-Text decoder, we propose a novel zero-shot plug-and-play framework and evaluate it on a challenging downstream task, EEG-based sentiment classification, where the task is to predict the sentence's sentiment based solely on the EEG signals. We show that our zero-shot framework can outperform fully-supervised baselines with a large margin. The code is made publicly available for research purpose at https://github.com/MikeWangWZHL/EEG-To-Text.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Wang, Zhenhailong

Contributors dc:contributor

• Ji, Heng

• Large Language Models
• Brain-to-text Decoding
• Multimodal Transfer Learning

Statement dc:rights

• Copyright 2023 Zhenhailong Wang

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/120213

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F120213