Fusing multimodal neural networks: a study on sleep classification and sound event localization and detection

This is an explorative study of multimodal large-scale transformer networks. The thesis explores methods to pretrain and fuse multiple large-scale transformer networks each responsible for a modality, in order to improve their performance on different tasks. Specifically, we first dive into the task of infant sleep classification using audio, electrocardiogram (ECG), and inertial measurement unit (IMU). We explore various pretraining and finetuning schemes, as well as different fusion techniques. We also assess the effectiveness of fusion by cross-attention with sound event localization and detection (SELD), a multichannel machine learning task with multiple outputs. We show that this multimodal network structure is generic enough to work in various settings.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Chang, Kai Chieh

Contributors dc:contributor

• Hasegawa-Johnson, Mark

• Multimodal
• Machine Learning

Statement dc:rights

• Copyright 2024 Kai Chieh Chang

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124593

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124593