Language-independent methods for computer-assisted pronunciation training

Computer-assisted pronunciation training (CAPT) systems help students practice speaking foreign languages by providing automatic pronunciation assessment and corrective feedback. Automatic speech recognition (ASR) technology is a natural component in CAPT systems. Since a nonnative speaker's native language (Li) background affects their pronunciation patterns in a target language (L2), typically not only native but also nonnative training data of specific Ls is needed to train a recognizer for CAPT systems. Given that there are around 7,000 languages in the world, the data collection process is costly and has scalability issues. In addition, expert knowledge on the target L2 is also often needed to design a large feature set describing the deviation of nonnative speech from native speech. In contrast to machines, it is relatively easy for native listeners to detect pronunciation errors without being exposed to nonnative speech or trained with linguistic knowledge beforehand. In this thesis, we are interested in this unsupervised capability and propose methods to overcome the language-dependent challenges. Inspired by the success of unsupervised acoustic pattern discovery, we propose to discover an individual learner's pronunciation error patterns in an unsupervised manner by analyzing the acoustic similarity between speech segments from the learner. Experimental results on nonnative English and nonnative Mandarin Chinese spoken by students from different Ls show that the proposed method is Li-independent and can be portable to different L2s. Moreover, the method is personalized such that it accommodates variations in pronunciation patterns across students. In addition, motivated by the success of deep learning models in unsupervised feature learning, we explore the use of convolutional neural networks (CNNs) for mispronunciation detection. A language-independent data augmentation method is developed to take advantage of native speech as training samples. Experimental results on nonnative Mandarin Chinese speech show the effectiveness of the model and the method. Moreover, both qualitative and quantitative analyses on the convolutional filters reveal that the CNN automatically learns a set of human-interpretable high-level features.

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2016

Author dc:creator

• Lee, Ann, Ph. D. Massachusetts Institute of Technology

Advisor dc:contributor.advisor

• James Glass.

• Electrical Engineering and Computer Science.

Statement dc:rights

• MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

http://hdl.handle.net/1721.1/107338

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/107338