{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/120213"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/120213","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Open-vocabulary brain-to-text decoding via cross-modal transfer with large language models","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","abstract_has_math":false,"creators":["Wang, Zhenhailong"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Ji, Heng"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2023,"date_issued":"2023-05","date_published":"2023-05","updated_at":"2026-07-22T22:24:57Z","subjects":["Large Language Models","Brain-to-text Decoding","Multimodal Transfer Learning"],"languages":["en","eng"],"rights":["Copyright 2023 Zhenhailong Wang"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/120213","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Ji, Heng"]},{"key":"dc:creator","label":"Author","values":["Wang, Zhenhailong"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2023-05","2023-03-13"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Large Language Models","Brain-to-text Decoding","Multimodal Transfer Learning"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2023 Zhenhailong Wang"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/120213"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","The student, Zhenhailong Wang, accepted the attached license on 2023-03-10 at 20:03.","The student, Zhenhailong Wang, submitted this Thesis for approval on 2023-03-10 at 20:10.","This Thesis was approved for publication on 2023-03-13 at 14:24.","DSpace SAF Submission Ingestion Package generated from Vireo submission #18871 on 2023-09-01 at 17:07:31","Recent research on transformer-based large language models and large-scale pretraining has shown unprecedented success in the field of natural language processing and beyond. A slew of previous work demonstrated that the rich knowledge embedded in large language models can be effective transferred across different modalities, to solve a wider range of discrinimative and generative tasks, e.g., vision-language understanding and robotic planning. In this thesis, we investigate an uncharted area of cross-modal transfer with large language models, decoding brain signals. State-of-the-art brain-to-text systems have achieved initial success in decoding language directly from brain signals using modern machine learning models such as recurrent neural networks. However, current approaches are limited to small closed vocabularies which are far from enough for natural communication. Additionally, most of the high-performing approaches require data from invasive devices (e.g., ECoG). To address the existing limitations, we extend the brain-to-text decoding problem to a more challenging open vocabulary setting, where the decoding is performed on the entire English vocabulary that is 100x larger than previous work. Furthermore, we decode from Electroencephalography (EEG) data which is non-invasive but more noisy. To handle the huge vocabulary size with a small amount of parallel EEG-text data, we propose a novel model leveraging pretrained large language models. The key idea is to learn a good brain-language alignment between the ``brain embedding space'' and the ``pretrained language embedding space'' so that we can directly leverage the power of a large language model, e.g., BART. Results show promising EEG-to-Text decoding performance with a BLEU-1 score of 40.1%. Moreover, in-depth analysis shows that the proposed model has strong scalability and robustness with respect to the data source, which is essential towards building a large-scale brain-language foundation model. To further demonstrate the effectiveness of the EEG-to-Text decoder, we propose a novel zero-shot plug-and-play framework and evaluate it on a challenging downstream task, EEG-based sentiment classification, where the task is to predict the sentence's sentiment based solely on the EEG signals. We show that our zero-shot framework can outperform fully-supervised baselines with a large margin. The code is made publicly available for research purpose at https://github.com/MikeWangWZHL/EEG-To-Text."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Open-vocabulary brain-to-text decoding via cross-modal transfer with large language models"]}]}],"canonical_facts":{"dc:contributor":["Ji, Heng"],"dc:creator":["Wang, Zhenhailong"],"dc:date":["2023-05","2023-03-13"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms","The student, Zhenhailong Wang, accepted the attached license on 2023-03-10 at 20:03.","The student, Zhenhailong Wang, submitted this Thesis for approval on 2023-03-10 at 20:10.","This Thesis was approved for publication on 2023-03-13 at 14:24.","DSpace SAF Submission Ingestion Package generated from Vireo submission #18871 on 2023-09-01 at 17:07:31","Recent research on transformer-based large language models and large-scale pretraining has shown unprecedented success in the field of natural language processing and beyond. A slew of previous work demonstrated that the rich knowledge embedded in large language models can be effective transferred across different modalities, to solve a wider range of discrinimative and generative tasks, e.g., vision-language understanding and robotic planning. In this thesis, we investigate an uncharted area of cross-modal transfer with large language models, decoding brain signals. State-of-the-art brain-to-text systems have achieved initial success in decoding language directly from brain signals using modern machine learning models such as recurrent neural networks. However, current approaches are limited to small closed vocabularies which are far from enough for natural communication. Additionally, most of the high-performing approaches require data from invasive devices (e.g., ECoG). To address the existing limitations, we extend the brain-to-text decoding problem to a more challenging open vocabulary setting, where the decoding is performed on the entire English vocabulary that is 100x larger than previous work. Furthermore, we decode from Electroencephalography (EEG) data which is non-invasive but more noisy. To handle the huge vocabulary size with a small amount of parallel EEG-text data, we propose a novel model leveraging pretrained large language models. The key idea is to learn a good brain-language alignment between the ``brain embedding space'' and the ``pretrained language embedding space'' so that we can directly leverage the power of a large language model, e.g., BART. Results show promising EEG-to-Text decoding performance with a BLEU-1 score of 40.1%. Moreover, in-depth analysis shows that the proposed model has strong scalability and robustness with respect to the data source, which is essential towards building a large-scale brain-language foundation model. To further demonstrate the effectiveness of the EEG-to-Text decoder, we propose a novel zero-shot plug-and-play framework and evaluate it on a challenging downstream task, EEG-based sentiment classification, where the task is to predict the sentence's sentiment based solely on the EEG signals. We show that our zero-shot framework can outperform fully-supervised baselines with a large margin. The code is made publicly available for research purpose at https://github.com/MikeWangWZHL/EEG-To-Text."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/120213"],"dc:language":["en","eng"],"dc:rights":["Copyright 2023 Zhenhailong Wang"],"dc:subject":["Large Language Models","Brain-to-text Decoding","Multimodal Transfer Learning"],"dc:title":["Open-vocabulary brain-to-text decoding via cross-modal transfer with large language models"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:57Z"}