Multimodal spoken unit discovery with paired and unpaired modalities

This thesis addresses the challenge of low-resource speech recognition by formulating it as a multimodal learning problem. The goal is to build a multimodal spoken unit discovery system that does not require any textual transcripts. Instead, it leverages speech and semantically related, multimodal signals such as paired images, unpaired text and unpaired sign language videos. To this end, this thesis proposes several novel algorithms based on neural networks and probabilistic graphical models. Further, it provides theoretical insights and empirical evidence to validate the efficacy of multimodal signals for spoken unit discovery.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Wang, Liming

Contributors dc:contributor

• Hasegawa-Johnson, Mark
• Smaragdis, Paris
• Schwing, Alexander
• Fleck, Margaret

• Acoustic Unit Discovery
• Low-resource Speech Recognition
• Unsupervised Speech Recognition
• Multimodal Learning
• Self-supervised Learning
• Language Acquisition

Statement dc:rights

• Copyright 2023 by Liming Wang

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/121497

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F121497