{"id":{"repo_id":"cambridge","oai_identifier":"oai:www.repository.cam.ac.uk:1810/358366"},"canonical_url":"https://search.dev.ndltd.org/etd/cambridge/oai:www.repository.cam.ac.uk:1810/358366","repository":{"repo_id":"cambridge","name":"Cambridge University","base_url":"https://api.repository.cam.ac.uk/server/oai/request"},"display":{"title":"End-to-end Contextual Speech Recognition and Understanding","abstract":"Contextual knowledge is of vital importance to end-to-end automatic speech recognition (ASR) and spoken language understanding systems (SLU), especially for the long-tailed word problem where systems suffer from degraded performance on rare or unseen words that are both relevant to the context and carrying important information. Integrating such contextual knowledge into such end-to-end systems is both necessary and challenging, as contextual knowledge is always dynamically changing while neural systems adopt a static set of trained parameters. In ASR, dynamic contextual knowledge is often incorporated via contextual biasing, where a list of rare words or phrases that are likely to appear in a given context is included, denoted as a biasing list of biasing words. A word is more likely to be correctly recognised if it is incorporated into the biasing list. The tree-constrained pointer generator (TCPGen) component, initially proposed for contextual biasing in ASR and later extended to end-to-end SLU systems, is introduced as the core of all the contributions in this thesis. TCPGen leverages the pointer generator mechanism to directly modify the model output distribution, which is both effective and has good generalisation ability. This combines the advantage of two widely-used biasing methods: the effectiveness of shallow fusion and the flexibility of neural deep biasing. Meanwhile, a word piece prefix tree is used to structure the biasing list, which allows TCPGen to handle large biasing lists containing thousands of words or entities. TCPGen, as a generic biasing component, can be integrated into both the attention-based encoder-decoder (AED) model and the neural transducer (NT) model. The minimum biasing word error (MBWE) training and biasing-word-driven language model discounting (BLMD) algorithms for TCPGen are also introduced for training and inference respectively. Experiments performed across three different datasets spanning domains of LibriSpeech audiobooks data, meeting conversation and spoken dialogue have shown consistent and significant word error rate (WER) improvements using TCPGen, especially on rare words. TCPGen also has the potential to be applied to universal large speech models via distribution-level adaptation, and the Whisper ASR model is used as an example to evaluate the effectiveness of TCPGen for this purpose. TCPGen can be further equipped with graph neural networks (GNN) by exploiting the tree structure of the biasing list. GNN encodings provide more powerful node representations in the prefix tree of TCPGen, allowing for \"lookahead\" functionality where each node contains not only its own word piece information but also information about its child branches. This improved node representation in TCPGen leads to a more accurate generation probability predictions for biasing words, enabling better determination of contextual biasing by incorporating information about future word pieces during each ASR decoding step. Three types of GNN encodings are explored in this thesis, including tree recursive neural networks (tree-RNN), graph convolutional networks (GCN) and GraphSAGE. Combinations of GCN and GraphSAGE that take advantage of the complementarity are also explored. To further illustrate the potential of applying contextual biasing and TCPGen in real-world scenarios with a particular focus on academic and educational applications, an audio-visual contextual ASR pipeline is also introduced using the augmented multi-party interaction (AMI) data. Experiments show further significant improvements using GNNs compared to the standard TCPGen on rare words on both the LibriSpeech data and on the audio-visual contextual ASR pipeline. Notably, the best-performing combination achieved a relative 60% WER reduction compared to the standard end-to-end ASR systems. As ASR, TCPGen can also be integrated into end-to-end SLU systems to boost the performance on rare words, especially for the slot-filling task. In SLU systems, a focused biasing list can be extracted from a structured knowledge base (KB) by predicting a slot shortlist (SS) using a class language model (CLM). The distribution over the valid set of word pieces can be converted into a distribution over all slot types, achieving a shortcut to the model slot output, termed slot probability biasing (SPB). Experiments on the combination of the two methods achieve significant performance improvements in slot filling, with a 1.7% absolute SLU-F1 increase overall and a 7% increase in unseen entities compared to a pipeline system. SPB further enabled zero-shot learning of unseen slot types by providing a list of possible named entities for that slot. Furthermore, a knowledge-aware audio-grounded (KA2G) generative SLU framework is introduced which performs slot filling by prompt and response in natural language. A stacked TCPGen component is incorporated with a shared tree encoding, such that knowledge explored during ASR beam search decoding can be transferred to the generation of slot values. The KA2G framework achieves further performance improvements compared to the SPB mechanism, especially on few-shot entities by exposing only a handful of examples to the model in the training set. Specifically, KA2G, compared to the end-to-end SLU with TCPGen, achieved a further 1.1\\% absolute increase in SLU-F1 overall, with an 8\\% increase on unseen entities.","abstract_html":"Contextual knowledge is of vital importance to end-to-end automatic speech recognition (ASR) and spoken language understanding systems (SLU), especially for the long-tailed word problem where systems suffer from degraded performance on rare or unseen words that are both relevant to the context and carrying important information. Integrating such contextual knowledge into such end-to-end systems is both necessary and challenging, as contextual knowledge is always dynamically changing while neural systems adopt a static set of trained parameters. In ASR, dynamic contextual knowledge is often incorporated via contextual biasing, where a list of rare words or phrases that are likely to appear in a given context is included, denoted as a biasing list of biasing words. A word is more likely to be correctly recognised if it is incorporated into the biasing list. The tree-constrained pointer generator (TCPGen) component, initially proposed for contextual biasing in ASR and later extended to end-to-end SLU systems, is introduced as the core of all the contributions in this thesis. TCPGen leverages the pointer generator mechanism to directly modify the model output distribution, which is both effective and has good generalisation ability. This combines the advantage of two widely-used biasing methods: the effectiveness of shallow fusion and the flexibility of neural deep biasing. Meanwhile, a word piece prefix tree is used to structure the biasing list, which allows TCPGen to handle large biasing lists containing thousands of words or entities. TCPGen, as a generic biasing component, can be integrated into both the attention-based encoder-decoder (AED) model and the neural transducer (NT) model. The minimum biasing word error (MBWE) training and biasing-word-driven language model discounting (BLMD) algorithms for TCPGen are also introduced for training and inference respectively. Experiments performed across three different datasets spanning domains of LibriSpeech audiobooks data, meeting conversation and spoken dialogue have shown consistent and significant word error rate (WER) improvements using TCPGen, especially on rare words. TCPGen also has the potential to be applied to universal large speech models via distribution-level adaptation, and the Whisper ASR model is used as an example to evaluate the effectiveness of TCPGen for this purpose. TCPGen can be further equipped with graph neural networks (GNN) by exploiting the tree structure of the biasing list. GNN encodings provide more powerful node representations in the prefix tree of TCPGen, allowing for "lookahead" functionality where each node contains not only its own word piece information but also information about its child branches. This improved node representation in TCPGen leads to a more accurate generation probability predictions for biasing words, enabling better determination of contextual biasing by incorporating information about future word pieces during each ASR decoding step. Three types of GNN encodings are explored in this thesis, including tree recursive neural networks (tree-RNN), graph convolutional networks (GCN) and GraphSAGE. Combinations of GCN and GraphSAGE that take advantage of the complementarity are also explored. To further illustrate the potential of applying contextual biasing and TCPGen in real-world scenarios with a particular focus on academic and educational applications, an audio-visual contextual ASR pipeline is also introduced using the augmented multi-party interaction (AMI) data. Experiments show further significant improvements using GNNs compared to the standard TCPGen on rare words on both the LibriSpeech data and on the audio-visual contextual ASR pipeline. Notably, the best-performing combination achieved a relative 60% WER reduction compared to the standard end-to-end ASR systems. As ASR, TCPGen can also be integrated into end-to-end SLU systems to boost the performance on rare words, especially for the slot-filling task. In SLU systems, a focused biasing list can be extracted from a structured knowledge base (KB) by predicting a slot shortlist (SS) using a class language model (CLM). The distribution over the valid set of word pieces can be converted into a distribution over all slot types, achieving a shortcut to the model slot output, termed slot probability biasing (SPB). Experiments on the combination of the two methods achieve significant performance improvements in slot filling, with a 1.7% absolute SLU-F1 increase overall and a 7% increase in unseen entities compared to a pipeline system. SPB further enabled zero-shot learning of unseen slot types by providing a list of possible named entities for that slot. Furthermore, a knowledge-aware audio-grounded (KA2G) generative SLU framework is introduced which performs slot filling by prompt and response in natural language. A stacked TCPGen component is incorporated with a shared tree encoding, such that knowledge explored during ASR beam search decoding can be transferred to the generation of slot values. The KA2G framework achieves further performance improvements compared to the SPB mechanism, especially on few-shot entities by exposing only a handful of examples to the model in the training set. Specifically, KA2G, compared to the end-to-end SLU with TCPGen, achieved a further 1.1\\% absolute increase in SLU-F1 overall, with an 8\\% increase on unseen entities.","abstract_has_math":false,"creators":["Sun, Guangzhi"],"institution":"University of Cambridge","degree_name":"Doctor of Philosophy (PhD)","degree_level":"Doctoral","degree_discipline":null,"degree_department":null,"school":null,"contributors":[],"advisors":["Woodland, Philip"],"committee_chairs":[],"committee_members":[],"year":2023,"date_issued":"2023-06-28","date_published":"2023-06-28","updated_at":"2026-07-22T22:24:20Z","subjects":["deep learning","end-to-end","machine learning","speech","speech recognition","spoken language understanding"],"languages":["eng"],"rights":[],"rights_urls":["https://apollo8-f-pro.lib.cam.ac.uk/bitstreams/e85ab875-c5a7-4c6c-b668-ed4be18c90e3/download","https://creativecommons.org/licenses/by/4.0/"],"identifier_entries":[]},"links":{"outbound_url":"https://doi.org/10.17863/CAM.102062","outbound_label":"DOI","outbound_source":"dc:identifier.doi"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Woodland, Philip"]},{"key":"dc:contributor.sponsor","label":"Sponsor","values":["Cambridge Trust"]},{"key":"dc:creator","label":"Author","values":["Sun, Guangzhi"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.issued","label":"Date","values":["2023-06-28"]},{"key":"dc:publisher.institution","label":"Dc Publisher Institution","values":["University of Cambridge"]},{"key":"dc:relation.isreferencedby.uri","label":"Dc Relation Isreferencedby URI","values":["https://www.repository.cam.ac.uk/handle/1810/358366"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"dc:type.qualificationlevel","label":"Dc Type Qualificationlevel","values":["Doctoral"]},{"key":"dc:type.qualificationname","label":"Dc Type Qualificationname","values":["Doctor of Philosophy (PhD)"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["deep learning","end-to-end","machine learning","speech","speech recognition","spoken language understanding"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["eng"]},{"key":"dc:rights","label":"Dc Rights","values":["https://apollo8-f-pro.lib.cam.ac.uk/bitstreams/e85ab875-c5a7-4c6c-b668-ed4be18c90e3/download","https://creativecommons.org/licenses/by/4.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.doi","label":"DOI","values":["https://doi.org/10.17863/CAM.102062"]},{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://apollo8-f-pro.lib.cam.ac.uk/bitstreams/e23d3d46-bffc-4f50-a029-e43ce720dc2a/download"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Contextual knowledge is of vital importance to end-to-end automatic speech recognition (ASR) and spoken language understanding systems (SLU), especially for the long-tailed word problem where systems suffer from degraded performance on rare or unseen words that are both relevant to the context and carrying important information. Integrating such contextual knowledge into such end-to-end systems is both necessary and challenging, as contextual knowledge is always dynamically changing while neural systems adopt a static set of trained parameters. In ASR, dynamic contextual knowledge is often incorporated via contextual biasing, where a list of rare words or phrases that are likely to appear in a given context is included, denoted as a biasing list of biasing words. A word is more likely to be correctly recognised if it is incorporated into the biasing list. The tree-constrained pointer generator (TCPGen) component, initially proposed for contextual biasing in ASR and later extended to end-to-end SLU systems, is introduced as the core of all the contributions in this thesis. TCPGen leverages the pointer generator mechanism to directly modify the model output distribution, which is both effective and has good generalisation ability. This combines the advantage of two widely-used biasing methods: the effectiveness of shallow fusion and the flexibility of neural deep biasing. Meanwhile, a word piece prefix tree is used to structure the biasing list, which allows TCPGen to handle large biasing lists containing thousands of words or entities. TCPGen, as a generic biasing component, can be integrated into both the attention-based encoder-decoder (AED) model and the neural transducer (NT) model. The minimum biasing word error (MBWE) training and biasing-word-driven language model discounting (BLMD) algorithms for TCPGen are also introduced for training and inference respectively. Experiments performed across three different datasets spanning domains of LibriSpeech audiobooks data, meeting conversation and spoken dialogue have shown consistent and significant word error rate (WER) improvements using TCPGen, especially on rare words. TCPGen also has the potential to be applied to universal large speech models via distribution-level adaptation, and the Whisper ASR model is used as an example to evaluate the effectiveness of TCPGen for this purpose. TCPGen can be further equipped with graph neural networks (GNN) by exploiting the tree structure of the biasing list. GNN encodings provide more powerful node representations in the prefix tree of TCPGen, allowing for \"lookahead\" functionality where each node contains not only its own word piece information but also information about its child branches. This improved node representation in TCPGen leads to a more accurate generation probability predictions for biasing words, enabling better determination of contextual biasing by incorporating information about future word pieces during each ASR decoding step. Three types of GNN encodings are explored in this thesis, including tree recursive neural networks (tree-RNN), graph convolutional networks (GCN) and GraphSAGE. Combinations of GCN and GraphSAGE that take advantage of the complementarity are also explored. To further illustrate the potential of applying contextual biasing and TCPGen in real-world scenarios with a particular focus on academic and educational applications, an audio-visual contextual ASR pipeline is also introduced using the augmented multi-party interaction (AMI) data. Experiments show further significant improvements using GNNs compared to the standard TCPGen on rare words on both the LibriSpeech data and on the audio-visual contextual ASR pipeline. Notably, the best-performing combination achieved a relative 60% WER reduction compared to the standard end-to-end ASR systems. As ASR, TCPGen can also be integrated into end-to-end SLU systems to boost the performance on rare words, especially for the slot-filling task. In SLU systems, a focused biasing list can be extracted from a structured knowledge base (KB) by predicting a slot shortlist (SS) using a class language model (CLM). The distribution over the valid set of word pieces can be converted into a distribution over all slot types, achieving a shortcut to the model slot output, termed slot probability biasing (SPB). Experiments on the combination of the two methods achieve significant performance improvements in slot filling, with a 1.7% absolute SLU-F1 increase overall and a 7% increase in unseen entities compared to a pipeline system. SPB further enabled zero-shot learning of unseen slot types by providing a list of possible named entities for that slot. Furthermore, a knowledge-aware audio-grounded (KA2G) generative SLU framework is introduced which performs slot filling by prompt and response in natural language. A stacked TCPGen component is incorporated with a shared tree encoding, such that knowledge explored during ASR beam search decoding can be transferred to the generation of slot values. The KA2G framework achieves further performance improvements compared to the SPB mechanism, especially on few-shot entities by exposing only a handful of examples to the model in the training set. Specifically, KA2G, compared to the end-to-end SLU with TCPGen, achieved a further 1.1\\% absolute increase in SLU-F1 overall, with an 8\\% increase on unseen entities."]},{"key":"dc:format.checksum.md5","label":"Dc Format Checksum Md5","values":["b2379c0ff4ee0638be45fef425d320bc","87eda9de84448d1f82354d60eee3eb5f"]},{"key":"dc:title","label":"Title","values":["End-to-end Contextual Speech Recognition and Understanding"]}]}],"canonical_facts":{"dc:contributor.advisor":["Woodland, Philip"],"dc:contributor.sponsor":["Cambridge Trust"],"dc:creator":["Sun, Guangzhi"],"dc:date.issued":["2023-06-28"],"dc:description.abstract":["Contextual knowledge is of vital importance to end-to-end automatic speech recognition (ASR) and spoken language understanding systems (SLU), especially for the long-tailed word problem where systems suffer from degraded performance on rare or unseen words that are both relevant to the context and carrying important information. Integrating such contextual knowledge into such end-to-end systems is both necessary and challenging, as contextual knowledge is always dynamically changing while neural systems adopt a static set of trained parameters. In ASR, dynamic contextual knowledge is often incorporated via contextual biasing, where a list of rare words or phrases that are likely to appear in a given context is included, denoted as a biasing list of biasing words. A word is more likely to be correctly recognised if it is incorporated into the biasing list. The tree-constrained pointer generator (TCPGen) component, initially proposed for contextual biasing in ASR and later extended to end-to-end SLU systems, is introduced as the core of all the contributions in this thesis. TCPGen leverages the pointer generator mechanism to directly modify the model output distribution, which is both effective and has good generalisation ability. This combines the advantage of two widely-used biasing methods: the effectiveness of shallow fusion and the flexibility of neural deep biasing. Meanwhile, a word piece prefix tree is used to structure the biasing list, which allows TCPGen to handle large biasing lists containing thousands of words or entities. TCPGen, as a generic biasing component, can be integrated into both the attention-based encoder-decoder (AED) model and the neural transducer (NT) model. The minimum biasing word error (MBWE) training and biasing-word-driven language model discounting (BLMD) algorithms for TCPGen are also introduced for training and inference respectively. Experiments performed across three different datasets spanning domains of LibriSpeech audiobooks data, meeting conversation and spoken dialogue have shown consistent and significant word error rate (WER) improvements using TCPGen, especially on rare words. TCPGen also has the potential to be applied to universal large speech models via distribution-level adaptation, and the Whisper ASR model is used as an example to evaluate the effectiveness of TCPGen for this purpose. TCPGen can be further equipped with graph neural networks (GNN) by exploiting the tree structure of the biasing list. GNN encodings provide more powerful node representations in the prefix tree of TCPGen, allowing for \"lookahead\" functionality where each node contains not only its own word piece information but also information about its child branches. This improved node representation in TCPGen leads to a more accurate generation probability predictions for biasing words, enabling better determination of contextual biasing by incorporating information about future word pieces during each ASR decoding step. Three types of GNN encodings are explored in this thesis, including tree recursive neural networks (tree-RNN), graph convolutional networks (GCN) and GraphSAGE. Combinations of GCN and GraphSAGE that take advantage of the complementarity are also explored. To further illustrate the potential of applying contextual biasing and TCPGen in real-world scenarios with a particular focus on academic and educational applications, an audio-visual contextual ASR pipeline is also introduced using the augmented multi-party interaction (AMI) data. Experiments show further significant improvements using GNNs compared to the standard TCPGen on rare words on both the LibriSpeech data and on the audio-visual contextual ASR pipeline. Notably, the best-performing combination achieved a relative 60% WER reduction compared to the standard end-to-end ASR systems. As ASR, TCPGen can also be integrated into end-to-end SLU systems to boost the performance on rare words, especially for the slot-filling task. In SLU systems, a focused biasing list can be extracted from a structured knowledge base (KB) by predicting a slot shortlist (SS) using a class language model (CLM). The distribution over the valid set of word pieces can be converted into a distribution over all slot types, achieving a shortcut to the model slot output, termed slot probability biasing (SPB). Experiments on the combination of the two methods achieve significant performance improvements in slot filling, with a 1.7% absolute SLU-F1 increase overall and a 7% increase in unseen entities compared to a pipeline system. SPB further enabled zero-shot learning of unseen slot types by providing a list of possible named entities for that slot. Furthermore, a knowledge-aware audio-grounded (KA2G) generative SLU framework is introduced which performs slot filling by prompt and response in natural language. A stacked TCPGen component is incorporated with a shared tree encoding, such that knowledge explored during ASR beam search decoding can be transferred to the generation of slot values. The KA2G framework achieves further performance improvements compared to the SPB mechanism, especially on few-shot entities by exposing only a handful of examples to the model in the training set. Specifically, KA2G, compared to the end-to-end SLU with TCPGen, achieved a further 1.1\\% absolute increase in SLU-F1 overall, with an 8\\% increase on unseen entities."],"dc:format.checksum.md5":["b2379c0ff4ee0638be45fef425d320bc","87eda9de84448d1f82354d60eee3eb5f"],"dc:identifier.doi":["https://doi.org/10.17863/CAM.102062"],"dc:identifier.uri":["https://apollo8-f-pro.lib.cam.ac.uk/bitstreams/e23d3d46-bffc-4f50-a029-e43ce720dc2a/download"],"dc:language":["eng"],"dc:publisher.institution":["University of Cambridge"],"dc:relation.isreferencedby.uri":["https://www.repository.cam.ac.uk/handle/1810/358366"],"dc:rights":["https://apollo8-f-pro.lib.cam.ac.uk/bitstreams/e85ab875-c5a7-4c6c-b668-ed4be18c90e3/download","https://creativecommons.org/licenses/by/4.0/"],"dc:subject":["deep learning","end-to-end","machine learning","speech","speech recognition","spoken language understanding"],"dc:title":["End-to-end Contextual Speech Recognition and Understanding"],"dc:type":["Thesis"],"dc:type.qualificationlevel":["Doctoral"],"dc:type.qualificationname":["Doctor of Philosophy (PhD)"]},"updated_at":"2026-07-22T22:24:20Z"}