Unsupervised learning of cross-modal mappings between speech and text

Deep learning is one of the most prominent machine learning techniques nowadays, being the state-of-the-art on a broad range of applications in computer vision, natural language processing, and speech and audio processing. Current deep learning models, however, rely on signicant amounts of supervision for training to achieve exceptional performance. For example, commercial speech recognition systems are usually trained on tens of thousands of hours of annotated data, which take the form of audio paired with transcriptions for training acoustic models, collections of text for training language models, and (possibly) linguist-crafted lexicons mapping words to their pronunciations. The immense cost of collecting these resources makes applying state-of-the-art speech recognition algorithm to under-resourced languages infeasible. In this thesis, we propose a general framework for mapping sequences between speech and text. Each component in this framework can be trained without any labeled data so the entire framework is unsupervised. We first propose a novel neural architecture that learns to represent a spoken word in an unlabeled speech corpus as an embedding vector in a latent space, in which word semantics and relationships between words are captured. In parallel, we train another latent space that captures similar information about written words using a corpus of unannotated text. By exploiting the geometrical properties exhibited in the speech and text embedding spaces, we develop an unsupervised learning algorithm that learns a cross-modal alignment between speech and text. As an example application of the learned alignment, we develop a unsupervised speech-to-text translation system using only unlabeled speech and text corpora.

Name thesis:degree_name

Master

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2019

Author dc:creator

• Chung, Yu-An.

Advisor dc:contributor.advisor

• James R. Glass.

• Electrical Engineering and Computer Science.

Statement dc:rights

• MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/122695

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/122695