Explainable artificial intelligence for inclusive automatic speech recognition

While the widespread adoption of automatic speech recognition (ASR) technology has brought significant benefits to society, it has also highlighted a persistent issue of inequality in access and utilization of technology. Furthermore, in response to the increasing prevalence of artificial intelligence applications, there has been a growing demand for explainable artificial intelligence (XAI). To address the need for interpretability and explainability in ASR, particularly in the context of inclusiveness, this paper aims to visualize the inner workings of the convolutional neural network (CNN) layer and Transformer block in Wav2Vec2.0. This is achieved by calculating the weighted relevance of the connectionist temporal classification (CTC) with respect to the attention and convolutional layers. Leveraging a Wav2Vec2.0 model pre-trained and fine-tuned on LibriSpeech, and testing the model using the Speech Accent Archive, we discovered that the Transformer exhibits a focus on other vowel transcriptions when encountering vowels within a word, whereas it exhibits a more localized attention when transcribing consonants or vowels in non-words absent from its learned vocabulary. Analysis of the weighted convolutional relevance in the first layer of the CNN revealed that different channels concentrate on distinct frequency and time sequences to capture the overall input characteristics. By obtaining a comprehensive understanding of the underlying causes and dynamics behind performance disparities, we can strive to mitigate these disparities and promote a more inclusive ASR technology.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Lee, Seunghyun

Contributors dc:contributor

• Hasegawa-Johnson, Mark A

• Inclusive Asr
• Explainable Ai
• Asr Visualization

Statement dc:rights

• Copyright 2023 Seunghyun Lee

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/121386

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F121386