{"id":{"repo_id":"cambridge","oai_identifier":"oai:www.repository.cam.ac.uk:1810/396905"},"canonical_url":"https://search.dev.ndltd.org/etd/cambridge/oai:www.repository.cam.ac.uk:1810/396905","repository":{"repo_id":"cambridge","name":"Cambridge University","base_url":"https://api.repository.cam.ac.uk/server/oai/request"},"display":{"title":"Neural Time Alignment for End-to-End Automatic Speech Recognition Systems","abstract":"Automatic Speech Recognition (ASR) is an important component for machines to interact with humans. With the development of deep learning, ASR systems can obtain very low error rates when there is a lot of training data available. Apart from the recognised text, it is also important to determine when a certain part of the text is spoken in the speech audio, which requires a time alignment between the text and audio to be generated. Conventional ASR systems are based on Hidden Markov Models (HMMs), and use the Viterbi algorithm for generating time alignments for the ASR system output. These time alignments are empirically accurate and can be used as the reference when human labels are not available. Nowadays end-to-end trainable (E2E) ASR systems have become the mainstream in ASR research and applications. Unlike conventional HMM-based systems, E2E systems cannot provide reliable word-level time alignments by default, and this thesis focuses on new approaches for obtaining reliable and accurate time alignments for E2E ASR systems. There are different types of E2E ASR systems that operate in rather different ways, including Neural Transducers (NTs) and Attention-based Encoder-Decoder (AED) models. With the development of speech foundation models, the ASR models routinely used in applications can be very large (with perhaps billions of parameters) and very expensive to train. Hence the provision of a light-weight add-on time alignment module that can be applied to different trained ASR backbone models without retraining ASR is a desirable solution. In this thesis, the Label-Synchronous Neural Aligner (LSyncNA) module is proposed. The LSyncNA module is based on neural networks and takes vectors from the ASR encoder and decoder, therefore it can be easily integrated into different E2E ASR models. Instead of directly finding the absolute word time boundaries, LSyncNA models the normalised duration of words and silence in an utterance, which can be converted to absolute time by scaling using the actual total time of the utterance. In this way, time alignment generation using LSyncNA only requires a forward pass through the neural network which is very fast and can be easily parallelised. Experiments on the LibriSpeech and TIMIT datasets using different ASR backbones showed that the LSyncNA module can generate accurate word-level time alignment for utterances of moderate lengths overall (no longer than 10 s), with an absolute timestamp error about 50 ms to 60 ms. In general, discriminative modelling is used in E2E ASR systems and generative modelling is used for conventional HMM-based ASR. The HMM emission probabilities are used to generate the acoustic probability for Viterbi time alignment, while only posterior probabilities of output tokens are available for E2E ASR models. Therefore the Frame-Synchronous Neural Aligner (FSyncNA) is proposed for E2E ASR to imitate the Viterbi alignment process using the estimated equivalent acoustic probabilities. Neural networks are used to model the acoustic probabilities using the vectors from the ASR encoder and decoder as the input, and deal with Voice Activity Detection (VAD). Unlike the LSyncNA module, the FSyncNA module uses dynamic programming (the Viterbi algorithm) to find out the absolute word-level time alignment, hence the inference phase of FSyncNA is slower than LSyncNA. Experiments on the LibriSpeech and TIMIT datasets using different ASR backbones showed that the FSyncNA module can generate more accurate timestamps compared to the LSyncNA module, and is not limited by the length of the utterance. The average timestamp error can be as low as 20 ms using the FSyncNA module. The proposed LSyncNA and FSyncNA were then tested using the Whisper universal ASR backbone, which has been widely used in ASR research and applications. Under relatively clean conditions (for LibriSpeech and TIMIT), the LSyncNA module is still applicable with minor adjustments. Since Whisper can be used for multilingual data, its output vocabulary is very large and the original FSyncNA formulation must be modified and optimised to reduce its memory footprint. Therefore a Memory-Efficient (ME) version of FSyncNA is proposed by focusing on the aligner hidden states which are much shorter vectors compared to the output probabilities. Compared to the original FSyncNA, ME-FSyncNA has a similar prediction accuracy with a smaller memory requirement, and achieves good performance under relatively clean conditions. To test for complex acoustic conditions with more natural speech, the Augmented Multi-party Interaction (AMI) corpus was used. Although Whisper is a universal ASR model, LSyncNA struggled with the acoustic complexity in AMI. The ME-FSyncNA showed its robustness but it had a poor performance in VAD. Hence adaptation was performed to the ME-FSyncNA VAD part using a mixture of LibriSpeech and AMI, and the performance was improved on both AMI and LibriSpeech. Finally, the ME-FSyncNA was tested on an ASR backbone that can output the transcription of multiple speakers, where overlapping speech could occur. With the oracle segmentations with respect to the speakers, very accurate word timestamps can be obtained from the ME-FSyncNA with an average error about 60 ms. But the word-level timestamp accuracy heavily relies on the time accuracy of the spoken utterance, and reducing the errors during text and speaker recognition in multispeaker audio is crucial for obtaining accurate word-level timestamps.","abstract_html":"Automatic Speech Recognition (ASR) is an important component for machines to interact with humans. With the development of deep learning, ASR systems can obtain very low error rates when there is a lot of training data available. Apart from the recognised text, it is also important to determine when a certain part of the text is spoken in the speech audio, which requires a time alignment between the text and audio to be generated. Conventional ASR systems are based on Hidden Markov Models (HMMs), and use the Viterbi algorithm for generating time alignments for the ASR system output. These time alignments are empirically accurate and can be used as the reference when human labels are not available. Nowadays end-to-end trainable (E2E) ASR systems have become the mainstream in ASR research and applications. Unlike conventional HMM-based systems, E2E systems cannot provide reliable word-level time alignments by default, and this thesis focuses on new approaches for obtaining reliable and accurate time alignments for E2E ASR systems. There are different types of E2E ASR systems that operate in rather different ways, including Neural Transducers (NTs) and Attention-based Encoder-Decoder (AED) models. With the development of speech foundation models, the ASR models routinely used in applications can be very large (with perhaps billions of parameters) and very expensive to train. Hence the provision of a light-weight add-on time alignment module that can be applied to different trained ASR backbone models without retraining ASR is a desirable solution. In this thesis, the Label-Synchronous Neural Aligner (LSyncNA) module is proposed. The LSyncNA module is based on neural networks and takes vectors from the ASR encoder and decoder, therefore it can be easily integrated into different E2E ASR models. Instead of directly finding the absolute word time boundaries, LSyncNA models the normalised duration of words and silence in an utterance, which can be converted to absolute time by scaling using the actual total time of the utterance. In this way, time alignment generation using LSyncNA only requires a forward pass through the neural network which is very fast and can be easily parallelised. Experiments on the LibriSpeech and TIMIT datasets using different ASR backbones showed that the LSyncNA module can generate accurate word-level time alignment for utterances of moderate lengths overall (no longer than 10 s), with an absolute timestamp error about 50 ms to 60 ms. In general, discriminative modelling is used in E2E ASR systems and generative modelling is used for conventional HMM-based ASR. The HMM emission probabilities are used to generate the acoustic probability for Viterbi time alignment, while only posterior probabilities of output tokens are available for E2E ASR models. Therefore the Frame-Synchronous Neural Aligner (FSyncNA) is proposed for E2E ASR to imitate the Viterbi alignment process using the estimated equivalent acoustic probabilities. Neural networks are used to model the acoustic probabilities using the vectors from the ASR encoder and decoder as the input, and deal with Voice Activity Detection (VAD). Unlike the LSyncNA module, the FSyncNA module uses dynamic programming (the Viterbi algorithm) to find out the absolute word-level time alignment, hence the inference phase of FSyncNA is slower than LSyncNA. Experiments on the LibriSpeech and TIMIT datasets using different ASR backbones showed that the FSyncNA module can generate more accurate timestamps compared to the LSyncNA module, and is not limited by the length of the utterance. The average timestamp error can be as low as 20 ms using the FSyncNA module. The proposed LSyncNA and FSyncNA were then tested using the Whisper universal ASR backbone, which has been widely used in ASR research and applications. Under relatively clean conditions (for LibriSpeech and TIMIT), the LSyncNA module is still applicable with minor adjustments. Since Whisper can be used for multilingual data, its output vocabulary is very large and the original FSyncNA formulation must be modified and optimised to reduce its memory footprint. Therefore a Memory-Efficient (ME) version of FSyncNA is proposed by focusing on the aligner hidden states which are much shorter vectors compared to the output probabilities. Compared to the original FSyncNA, ME-FSyncNA has a similar prediction accuracy with a smaller memory requirement, and achieves good performance under relatively clean conditions. To test for complex acoustic conditions with more natural speech, the Augmented Multi-party Interaction (AMI) corpus was used. Although Whisper is a universal ASR model, LSyncNA struggled with the acoustic complexity in AMI. The ME-FSyncNA showed its robustness but it had a poor performance in VAD. Hence adaptation was performed to the ME-FSyncNA VAD part using a mixture of LibriSpeech and AMI, and the performance was improved on both AMI and LibriSpeech. Finally, the ME-FSyncNA was tested on an ASR backbone that can output the transcription of multiple speakers, where overlapping speech could occur. With the oracle segmentations with respect to the speakers, very accurate word timestamps can be obtained from the ME-FSyncNA with an average error about 60 ms. But the word-level timestamp accuracy heavily relies on the time accuracy of the spoken utterance, and reducing the errors during text and speaker recognition in multispeaker audio is crucial for obtaining accurate word-level timestamps.","abstract_has_math":false,"creators":["Jiang, Dongcheng"],"institution":"University of Cambridge","degree_name":"Doctor of Philosophy (PhD)","degree_level":"Doctoral","degree_discipline":null,"degree_department":null,"school":null,"contributors":[],"advisors":["Woodland, Philip"],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025-09-15","date_published":"2025-09-15","updated_at":"2026-07-22T22:24:27Z","subjects":["automatic speech recognition","time alignment","deep learning"],"languages":["eng"],"rights":[],"rights_urls":["https://www.repository.cam.ac.uk/bitstreams/4bc0abb7-b423-4fa4-8913-8511f578df3c/download","https://creativecommons.org/licenses/by-nc/4.0/"],"identifier_entries":[]},"links":{"outbound_url":"https://doi.org/10.17863/CAM.126151","outbound_label":"DOI","outbound_source":"dc:identifier.doi"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Woodland, Philip"]},{"key":"dc:contributor.sponsor","label":"Sponsor","values":["Trinity Internal Graduate Studentship"]},{"key":"dc:creator","label":"Author","values":["Jiang, Dongcheng"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.issued","label":"Date","values":["2025-09-15"]},{"key":"dc:publisher.institution","label":"Dc Publisher Institution","values":["University of Cambridge"]},{"key":"dc:relation.isreferencedby.uri","label":"Dc Relation Isreferencedby URI","values":["https://www.repository.cam.ac.uk/handle/1810/396905"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"dc:type.qualificationlevel","label":"Dc Type Qualificationlevel","values":["Doctoral"]},{"key":"dc:type.qualificationname","label":"Dc Type Qualificationname","values":["Doctor of Philosophy (PhD)"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["automatic speech recognition","time alignment","deep learning"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["eng"]},{"key":"dc:rights","label":"Dc Rights","values":["https://www.repository.cam.ac.uk/bitstreams/4bc0abb7-b423-4fa4-8913-8511f578df3c/download","https://creativecommons.org/licenses/by-nc/4.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.doi","label":"DOI","values":["https://doi.org/10.17863/CAM.126151"]},{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://www.repository.cam.ac.uk/bitstreams/b25f62cd-f3e9-40df-bea9-a1cb75867ea3/download"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Automatic Speech Recognition (ASR) is an important component for machines to interact with humans. With the development of deep learning, ASR systems can obtain very low error rates when there is a lot of training data available. Apart from the recognised text, it is also important to determine when a certain part of the text is spoken in the speech audio, which requires a time alignment between the text and audio to be generated. Conventional ASR systems are based on Hidden Markov Models (HMMs), and use the Viterbi algorithm for generating time alignments for the ASR system output. These time alignments are empirically accurate and can be used as the reference when human labels are not available. Nowadays end-to-end trainable (E2E) ASR systems have become the mainstream in ASR research and applications. Unlike conventional HMM-based systems, E2E systems cannot provide reliable word-level time alignments by default, and this thesis focuses on new approaches for obtaining reliable and accurate time alignments for E2E ASR systems. There are different types of E2E ASR systems that operate in rather different ways, including Neural Transducers (NTs) and Attention-based Encoder-Decoder (AED) models. With the development of speech foundation models, the ASR models routinely used in applications can be very large (with perhaps billions of parameters) and very expensive to train. Hence the provision of a light-weight add-on time alignment module that can be applied to different trained ASR backbone models without retraining ASR is a desirable solution. In this thesis, the Label-Synchronous Neural Aligner (LSyncNA) module is proposed. The LSyncNA module is based on neural networks and takes vectors from the ASR encoder and decoder, therefore it can be easily integrated into different E2E ASR models. Instead of directly finding the absolute word time boundaries, LSyncNA models the normalised duration of words and silence in an utterance, which can be converted to absolute time by scaling using the actual total time of the utterance. In this way, time alignment generation using LSyncNA only requires a forward pass through the neural network which is very fast and can be easily parallelised. Experiments on the LibriSpeech and TIMIT datasets using different ASR backbones showed that the LSyncNA module can generate accurate word-level time alignment for utterances of moderate lengths overall (no longer than 10 s), with an absolute timestamp error about 50 ms to 60 ms. In general, discriminative modelling is used in E2E ASR systems and generative modelling is used for conventional HMM-based ASR. The HMM emission probabilities are used to generate the acoustic probability for Viterbi time alignment, while only posterior probabilities of output tokens are available for E2E ASR models. Therefore the Frame-Synchronous Neural Aligner (FSyncNA) is proposed for E2E ASR to imitate the Viterbi alignment process using the estimated equivalent acoustic probabilities. Neural networks are used to model the acoustic probabilities using the vectors from the ASR encoder and decoder as the input, and deal with Voice Activity Detection (VAD). Unlike the LSyncNA module, the FSyncNA module uses dynamic programming (the Viterbi algorithm) to find out the absolute word-level time alignment, hence the inference phase of FSyncNA is slower than LSyncNA. Experiments on the LibriSpeech and TIMIT datasets using different ASR backbones showed that the FSyncNA module can generate more accurate timestamps compared to the LSyncNA module, and is not limited by the length of the utterance. The average timestamp error can be as low as 20 ms using the FSyncNA module. The proposed LSyncNA and FSyncNA were then tested using the Whisper universal ASR backbone, which has been widely used in ASR research and applications. Under relatively clean conditions (for LibriSpeech and TIMIT), the LSyncNA module is still applicable with minor adjustments. Since Whisper can be used for multilingual data, its output vocabulary is very large and the original FSyncNA formulation must be modified and optimised to reduce its memory footprint. Therefore a Memory-Efficient (ME) version of FSyncNA is proposed by focusing on the aligner hidden states which are much shorter vectors compared to the output probabilities. Compared to the original FSyncNA, ME-FSyncNA has a similar prediction accuracy with a smaller memory requirement, and achieves good performance under relatively clean conditions. To test for complex acoustic conditions with more natural speech, the Augmented Multi-party Interaction (AMI) corpus was used. Although Whisper is a universal ASR model, LSyncNA struggled with the acoustic complexity in AMI. The ME-FSyncNA showed its robustness but it had a poor performance in VAD. Hence adaptation was performed to the ME-FSyncNA VAD part using a mixture of LibriSpeech and AMI, and the performance was improved on both AMI and LibriSpeech. Finally, the ME-FSyncNA was tested on an ASR backbone that can output the transcription of multiple speakers, where overlapping speech could occur. With the oracle segmentations with respect to the speakers, very accurate word timestamps can be obtained from the ME-FSyncNA with an average error about 60 ms. But the word-level timestamp accuracy heavily relies on the time accuracy of the spoken utterance, and reducing the errors during text and speaker recognition in multispeaker audio is crucial for obtaining accurate word-level timestamps."]},{"key":"dc:format.checksum.md5","label":"Dc Format Checksum Md5","values":["3c613885206798efc9ed12902d6652e0","87eda9de84448d1f82354d60eee3eb5f"]},{"key":"dc:title","label":"Title","values":["Neural Time Alignment for End-to-End Automatic Speech Recognition Systems"]}]}],"canonical_facts":{"dc:contributor.advisor":["Woodland, Philip"],"dc:contributor.sponsor":["Trinity Internal Graduate Studentship"],"dc:creator":["Jiang, Dongcheng"],"dc:date.issued":["2025-09-15"],"dc:description.abstract":["Automatic Speech Recognition (ASR) is an important component for machines to interact with humans. With the development of deep learning, ASR systems can obtain very low error rates when there is a lot of training data available. Apart from the recognised text, it is also important to determine when a certain part of the text is spoken in the speech audio, which requires a time alignment between the text and audio to be generated. Conventional ASR systems are based on Hidden Markov Models (HMMs), and use the Viterbi algorithm for generating time alignments for the ASR system output. These time alignments are empirically accurate and can be used as the reference when human labels are not available. Nowadays end-to-end trainable (E2E) ASR systems have become the mainstream in ASR research and applications. Unlike conventional HMM-based systems, E2E systems cannot provide reliable word-level time alignments by default, and this thesis focuses on new approaches for obtaining reliable and accurate time alignments for E2E ASR systems. There are different types of E2E ASR systems that operate in rather different ways, including Neural Transducers (NTs) and Attention-based Encoder-Decoder (AED) models. With the development of speech foundation models, the ASR models routinely used in applications can be very large (with perhaps billions of parameters) and very expensive to train. Hence the provision of a light-weight add-on time alignment module that can be applied to different trained ASR backbone models without retraining ASR is a desirable solution. In this thesis, the Label-Synchronous Neural Aligner (LSyncNA) module is proposed. The LSyncNA module is based on neural networks and takes vectors from the ASR encoder and decoder, therefore it can be easily integrated into different E2E ASR models. Instead of directly finding the absolute word time boundaries, LSyncNA models the normalised duration of words and silence in an utterance, which can be converted to absolute time by scaling using the actual total time of the utterance. In this way, time alignment generation using LSyncNA only requires a forward pass through the neural network which is very fast and can be easily parallelised. Experiments on the LibriSpeech and TIMIT datasets using different ASR backbones showed that the LSyncNA module can generate accurate word-level time alignment for utterances of moderate lengths overall (no longer than 10 s), with an absolute timestamp error about 50 ms to 60 ms. In general, discriminative modelling is used in E2E ASR systems and generative modelling is used for conventional HMM-based ASR. The HMM emission probabilities are used to generate the acoustic probability for Viterbi time alignment, while only posterior probabilities of output tokens are available for E2E ASR models. Therefore the Frame-Synchronous Neural Aligner (FSyncNA) is proposed for E2E ASR to imitate the Viterbi alignment process using the estimated equivalent acoustic probabilities. Neural networks are used to model the acoustic probabilities using the vectors from the ASR encoder and decoder as the input, and deal with Voice Activity Detection (VAD). Unlike the LSyncNA module, the FSyncNA module uses dynamic programming (the Viterbi algorithm) to find out the absolute word-level time alignment, hence the inference phase of FSyncNA is slower than LSyncNA. Experiments on the LibriSpeech and TIMIT datasets using different ASR backbones showed that the FSyncNA module can generate more accurate timestamps compared to the LSyncNA module, and is not limited by the length of the utterance. The average timestamp error can be as low as 20 ms using the FSyncNA module. The proposed LSyncNA and FSyncNA were then tested using the Whisper universal ASR backbone, which has been widely used in ASR research and applications. Under relatively clean conditions (for LibriSpeech and TIMIT), the LSyncNA module is still applicable with minor adjustments. Since Whisper can be used for multilingual data, its output vocabulary is very large and the original FSyncNA formulation must be modified and optimised to reduce its memory footprint. Therefore a Memory-Efficient (ME) version of FSyncNA is proposed by focusing on the aligner hidden states which are much shorter vectors compared to the output probabilities. Compared to the original FSyncNA, ME-FSyncNA has a similar prediction accuracy with a smaller memory requirement, and achieves good performance under relatively clean conditions. To test for complex acoustic conditions with more natural speech, the Augmented Multi-party Interaction (AMI) corpus was used. Although Whisper is a universal ASR model, LSyncNA struggled with the acoustic complexity in AMI. The ME-FSyncNA showed its robustness but it had a poor performance in VAD. Hence adaptation was performed to the ME-FSyncNA VAD part using a mixture of LibriSpeech and AMI, and the performance was improved on both AMI and LibriSpeech. Finally, the ME-FSyncNA was tested on an ASR backbone that can output the transcription of multiple speakers, where overlapping speech could occur. With the oracle segmentations with respect to the speakers, very accurate word timestamps can be obtained from the ME-FSyncNA with an average error about 60 ms. But the word-level timestamp accuracy heavily relies on the time accuracy of the spoken utterance, and reducing the errors during text and speaker recognition in multispeaker audio is crucial for obtaining accurate word-level timestamps."],"dc:format.checksum.md5":["3c613885206798efc9ed12902d6652e0","87eda9de84448d1f82354d60eee3eb5f"],"dc:identifier.doi":["https://doi.org/10.17863/CAM.126151"],"dc:identifier.uri":["https://www.repository.cam.ac.uk/bitstreams/b25f62cd-f3e9-40df-bea9-a1cb75867ea3/download"],"dc:language":["eng"],"dc:publisher.institution":["University of Cambridge"],"dc:relation.isreferencedby.uri":["https://www.repository.cam.ac.uk/handle/1810/396905"],"dc:rights":["https://www.repository.cam.ac.uk/bitstreams/4bc0abb7-b423-4fa4-8913-8511f578df3c/download","https://creativecommons.org/licenses/by-nc/4.0/"],"dc:subject":["automatic speech recognition","time alignment","deep learning"],"dc:title":["Neural Time Alignment for End-to-End Automatic Speech Recognition Systems"],"dc:type":["Thesis"],"dc:type.qualificationlevel":["Doctoral"],"dc:type.qualificationname":["Doctor of Philosophy (PhD)"]},"updated_at":"2026-07-22T22:24:27Z"}