{"id":{"repo_id":"cambridge","oai_identifier":"oai:www.repository.cam.ac.uk:1810/406210"},"canonical_url":"https://search.dev.ndltd.org/etd/cambridge/oai:www.repository.cam.ac.uk:1810/406210","repository":{"repo_id":"cambridge","name":"Cambridge University","base_url":"https://api.repository.cam.ac.uk/server/oai/request"},"display":{"title":"Speech Foundation Models for Audio Processing","abstract":"Foundation models are trained on massive amounts of data and can be adapted to support a wide range of downstream tasks. Their emergence has significantly shaped the development of natural language and speech technologies. Foundation speech models such as Whisper have shown strong performance across a variety of audio processing tasks, including automatic speech recognition (ASR) and speech translation. Unlike traditional systems that require task-specific architectures and extensive supervision, these models offer a unified and flexible framework that supports generalisation to new domains, languages, and task formulations. In this thesis, we present a comprehensive investigation into the capability, application, and vulnerability of foundation speech models in the context of audio processing, aiming to understand both their potential and limitations. This thesis begins by addressing ASR error correction, a task that aims to refine ASR outputs to improve the accuracy and readability of transcriptions. Traditional approaches typically take the 1-best ASR hypothesis as input, which restricts correction models to the limited context of a single transcription. In this work, we propose methods that leverage the ASR N-best hypotheses, allowing the error correction model to access a richer set of alternatives and make more informed predictions. To further guide the generation process, we introduce various constrained decoding strategies based on either the N-best list or an ASR lattice, enabling the error correction model to better utilise knowledge from the ASR system. Beyond supervised ASR error correction, we also investigate a training-free, zero-shot approach using large language models (LLMs). This method enables plug-and-play integration with existing ASR systems, without requiring access to model weights or additional training data. Experimental results across multiple test sets show that the proposed methods yield consistent performance improvements for two different ASR backends. Next, we investigate cross-lingual transfer in speech translation. Whisper is trained on multilingual speech data in a multitask setting, which suggests that utterances from different languages are projected into a shared semantic space by the acoustic encoder, thereby enabling cross-lingual transfer. To examine this property, we analyse encoder representations through a speech-to-speech retrieval task. Leveraging this shared representation, we extend Whisper’s speech translation capabilities to new source-target language pairs. By modifying the control tokens provided to the decoder, we enable zero-shot speech translation for language pairs not observed during Whisper’s pre-training. Fine-tuning Whisper on English-Chinese translation data further improves performance not only for English inputs but also for other languages, surpassing the zero-shot baseline. Finally, experiments on low-resource languages show that Whisper can translate speech from languages absent during pre-training, illustrating its strong generalisation ability through cross-lingual representations. We then focus on the use of Whisper for a more complex spoken language processing task: spoken grammatical error correction (SGEC), which aims to automatically detect and correct grammatical errors in transcribed speech. Traditional approaches typically rely on cascaded pipelines consisting of separate modules for ASR, disfluency detection, and grammatical error correction. However, such pipelines are prone to information loss and error propagation. To address these limitations, we propose an end-to-end framework that directly generates grammatically corrected text from speech input. To overcome the lack of training data, we develop a large-scale pseudo-labelling strategy that expands the corpus size substantially. Additional gains are achieved through contextual prompt-based learning and a novel alignment-based method that modifies training references to improve feedback precision. The proposed approach outperforms cascaded baseline systems and offers a practical solution for learner feedback applications. Foundation text and vision models have demonstrated emergent abilities to perform tasks beyond their pre-training objectives, a property that makes them especially valuable for real-world applications. In this thesis, we investigate Whisper’s emergent capability for zero-shot audio classification, a task outside its original training objective. We introduce a template-based prompting method that uses decoder probabilities for classification without requiring fine-tuning or labelled data. A simple unsupervised debiasing strategy further improves performance by reweighting class probabilities. The proposed method achieves strong results across eight benchmark datasets and surpasses previous zero-shot baselines. We further show that performance increases with model size, implying that as ASR foundation models scale up, they may exhibit improved zero-shot performance, akin to text-based foundation models. Finally, we address the potential vulnerability of foundation speech models to adversarial manipulation. We investigate universal acoustic adversarial attacks on speech LLMs, where a fixed adversarial audio segment is prepended to the input. This segment is learned using gradient-based optimisation on a small dataset and, once generated, can be reused for any speech input to that model without needing further access to the model or its prompt. In addition to general attacks, we introduce selective attacks that activate only under specific input conditions, such as a specific speaker gender or spoken language. These targeted attacks are more difficult to detect and raise serious concerns regarding fairness and safety. Experimental results show that the proposed attacks are highly effective against versatile speech LLMs such as Qwen2-Audio, highlighting the need for more robust training and defence mechanisms.","abstract_html":"Foundation models are trained on massive amounts of data and can be adapted to support a wide range of downstream tasks. Their emergence has significantly shaped the development of natural language and speech technologies. Foundation speech models such as Whisper have shown strong performance across a variety of audio processing tasks, including automatic speech recognition (ASR) and speech translation. Unlike traditional systems that require task-specific architectures and extensive supervision, these models offer a unified and flexible framework that supports generalisation to new domains, languages, and task formulations. In this thesis, we present a comprehensive investigation into the capability, application, and vulnerability of foundation speech models in the context of audio processing, aiming to understand both their potential and limitations. This thesis begins by addressing ASR error correction, a task that aims to refine ASR outputs to improve the accuracy and readability of transcriptions. Traditional approaches typically take the 1-best ASR hypothesis as input, which restricts correction models to the limited context of a single transcription. In this work, we propose methods that leverage the ASR N-best hypotheses, allowing the error correction model to access a richer set of alternatives and make more informed predictions. To further guide the generation process, we introduce various constrained decoding strategies based on either the N-best list or an ASR lattice, enabling the error correction model to better utilise knowledge from the ASR system. Beyond supervised ASR error correction, we also investigate a training-free, zero-shot approach using large language models (LLMs). This method enables plug-and-play integration with existing ASR systems, without requiring access to model weights or additional training data. Experimental results across multiple test sets show that the proposed methods yield consistent performance improvements for two different ASR backends. Next, we investigate cross-lingual transfer in speech translation. Whisper is trained on multilingual speech data in a multitask setting, which suggests that utterances from different languages are projected into a shared semantic space by the acoustic encoder, thereby enabling cross-lingual transfer. To examine this property, we analyse encoder representations through a speech-to-speech retrieval task. Leveraging this shared representation, we extend Whisper’s speech translation capabilities to new source-target language pairs. By modifying the control tokens provided to the decoder, we enable zero-shot speech translation for language pairs not observed during Whisper’s pre-training. Fine-tuning Whisper on English-Chinese translation data further improves performance not only for English inputs but also for other languages, surpassing the zero-shot baseline. Finally, experiments on low-resource languages show that Whisper can translate speech from languages absent during pre-training, illustrating its strong generalisation ability through cross-lingual representations. We then focus on the use of Whisper for a more complex spoken language processing task: spoken grammatical error correction (SGEC), which aims to automatically detect and correct grammatical errors in transcribed speech. Traditional approaches typically rely on cascaded pipelines consisting of separate modules for ASR, disfluency detection, and grammatical error correction. However, such pipelines are prone to information loss and error propagation. To address these limitations, we propose an end-to-end framework that directly generates grammatically corrected text from speech input. To overcome the lack of training data, we develop a large-scale pseudo-labelling strategy that expands the corpus size substantially. Additional gains are achieved through contextual prompt-based learning and a novel alignment-based method that modifies training references to improve feedback precision. The proposed approach outperforms cascaded baseline systems and offers a practical solution for learner feedback applications. Foundation text and vision models have demonstrated emergent abilities to perform tasks beyond their pre-training objectives, a property that makes them especially valuable for real-world applications. In this thesis, we investigate Whisper’s emergent capability for zero-shot audio classification, a task outside its original training objective. We introduce a template-based prompting method that uses decoder probabilities for classification without requiring fine-tuning or labelled data. A simple unsupervised debiasing strategy further improves performance by reweighting class probabilities. The proposed method achieves strong results across eight benchmark datasets and surpasses previous zero-shot baselines. We further show that performance increases with model size, implying that as ASR foundation models scale up, they may exhibit improved zero-shot performance, akin to text-based foundation models. Finally, we address the potential vulnerability of foundation speech models to adversarial manipulation. We investigate universal acoustic adversarial attacks on speech LLMs, where a fixed adversarial audio segment is prepended to the input. This segment is learned using gradient-based optimisation on a small dataset and, once generated, can be reused for any speech input to that model without needing further access to the model or its prompt. In addition to general attacks, we introduce selective attacks that activate only under specific input conditions, such as a specific speaker gender or spoken language. These targeted attacks are more difficult to detect and raise serious concerns regarding fairness and safety. Experimental results show that the proposed attacks are highly effective against versatile speech LLMs such as Qwen2-Audio, highlighting the need for more robust training and defence mechanisms.","abstract_has_math":false,"creators":["Ma, Rao"],"institution":"University of Cambridge","degree_name":"Doctor of Philosophy (PhD)","degree_level":"Doctoral","degree_discipline":null,"degree_department":null,"school":null,"contributors":[],"advisors":["Knill, katherine"],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025-09-02","date_published":"2025-09-02","updated_at":"2026-07-24T01:33:18Z","subjects":["audio processing","speech foundation models","automatic speech recognition","natural language processing","large language models"],"languages":["eng"],"rights":[],"rights_urls":["https://www.repository.cam.ac.uk/bitstreams/652a9f72-2d16-4812-90f1-492b87c83b1f/download","https://creativecommons.org/licenses/by/4.0/"],"identifier_entries":[]},"links":{"outbound_url":"https://doi.org/10.17863/CAM.132340","outbound_label":"DOI","outbound_source":"dc:identifier.doi"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Knill, katherine"]},{"key":"dc:contributor.sponsor","label":"Sponsor","values":["Cambridge University Press & Assessment Studentship"]},{"key":"dc:creator","label":"Author","values":["Ma, Rao"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.issued","label":"Date","values":["2025-09-02"]},{"key":"dc:publisher.institution","label":"Dc Publisher Institution","values":["University of Cambridge"]},{"key":"dc:relation.isreferencedby.uri","label":"Dc Relation Isreferencedby URI","values":["https://www.repository.cam.ac.uk/handle/1810/406210"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"dc:type.qualificationlevel","label":"Dc Type Qualificationlevel","values":["Doctoral"]},{"key":"dc:type.qualificationname","label":"Dc Type Qualificationname","values":["Doctor of Philosophy (PhD)"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["audio processing","speech foundation models","automatic speech recognition","natural language processing","large language models"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["eng"]},{"key":"dc:rights","label":"Dc Rights","values":["https://www.repository.cam.ac.uk/bitstreams/652a9f72-2d16-4812-90f1-492b87c83b1f/download","https://creativecommons.org/licenses/by/4.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.doi","label":"DOI","values":["https://doi.org/10.17863/CAM.132340"]},{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://www.repository.cam.ac.uk/bitstreams/2224e9e6-280f-4c0c-b69b-7d46b9bd1d0b/download"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Foundation models are trained on massive amounts of data and can be adapted to support a wide range of downstream tasks. Their emergence has significantly shaped the development of natural language and speech technologies. Foundation speech models such as Whisper have shown strong performance across a variety of audio processing tasks, including automatic speech recognition (ASR) and speech translation. Unlike traditional systems that require task-specific architectures and extensive supervision, these models offer a unified and flexible framework that supports generalisation to new domains, languages, and task formulations. In this thesis, we present a comprehensive investigation into the capability, application, and vulnerability of foundation speech models in the context of audio processing, aiming to understand both their potential and limitations. This thesis begins by addressing ASR error correction, a task that aims to refine ASR outputs to improve the accuracy and readability of transcriptions. Traditional approaches typically take the 1-best ASR hypothesis as input, which restricts correction models to the limited context of a single transcription. In this work, we propose methods that leverage the ASR N-best hypotheses, allowing the error correction model to access a richer set of alternatives and make more informed predictions. To further guide the generation process, we introduce various constrained decoding strategies based on either the N-best list or an ASR lattice, enabling the error correction model to better utilise knowledge from the ASR system. Beyond supervised ASR error correction, we also investigate a training-free, zero-shot approach using large language models (LLMs). This method enables plug-and-play integration with existing ASR systems, without requiring access to model weights or additional training data. Experimental results across multiple test sets show that the proposed methods yield consistent performance improvements for two different ASR backends. Next, we investigate cross-lingual transfer in speech translation. Whisper is trained on multilingual speech data in a multitask setting, which suggests that utterances from different languages are projected into a shared semantic space by the acoustic encoder, thereby enabling cross-lingual transfer. To examine this property, we analyse encoder representations through a speech-to-speech retrieval task. Leveraging this shared representation, we extend Whisper’s speech translation capabilities to new source-target language pairs. By modifying the control tokens provided to the decoder, we enable zero-shot speech translation for language pairs not observed during Whisper’s pre-training. Fine-tuning Whisper on English-Chinese translation data further improves performance not only for English inputs but also for other languages, surpassing the zero-shot baseline. Finally, experiments on low-resource languages show that Whisper can translate speech from languages absent during pre-training, illustrating its strong generalisation ability through cross-lingual representations. We then focus on the use of Whisper for a more complex spoken language processing task: spoken grammatical error correction (SGEC), which aims to automatically detect and correct grammatical errors in transcribed speech. Traditional approaches typically rely on cascaded pipelines consisting of separate modules for ASR, disfluency detection, and grammatical error correction. However, such pipelines are prone to information loss and error propagation. To address these limitations, we propose an end-to-end framework that directly generates grammatically corrected text from speech input. To overcome the lack of training data, we develop a large-scale pseudo-labelling strategy that expands the corpus size substantially. Additional gains are achieved through contextual prompt-based learning and a novel alignment-based method that modifies training references to improve feedback precision. The proposed approach outperforms cascaded baseline systems and offers a practical solution for learner feedback applications. Foundation text and vision models have demonstrated emergent abilities to perform tasks beyond their pre-training objectives, a property that makes them especially valuable for real-world applications. In this thesis, we investigate Whisper’s emergent capability for zero-shot audio classification, a task outside its original training objective. We introduce a template-based prompting method that uses decoder probabilities for classification without requiring fine-tuning or labelled data. A simple unsupervised debiasing strategy further improves performance by reweighting class probabilities. The proposed method achieves strong results across eight benchmark datasets and surpasses previous zero-shot baselines. We further show that performance increases with model size, implying that as ASR foundation models scale up, they may exhibit improved zero-shot performance, akin to text-based foundation models. Finally, we address the potential vulnerability of foundation speech models to adversarial manipulation. We investigate universal acoustic adversarial attacks on speech LLMs, where a fixed adversarial audio segment is prepended to the input. This segment is learned using gradient-based optimisation on a small dataset and, once generated, can be reused for any speech input to that model without needing further access to the model or its prompt. In addition to general attacks, we introduce selective attacks that activate only under specific input conditions, such as a specific speaker gender or spoken language. These targeted attacks are more difficult to detect and raise serious concerns regarding fairness and safety. Experimental results show that the proposed attacks are highly effective against versatile speech LLMs such as Qwen2-Audio, highlighting the need for more robust training and defence mechanisms."]},{"key":"dc:format.checksum.md5","label":"Dc Format Checksum Md5","values":["bef526f17496b2abea977b3ecbe02a73","87eda9de84448d1f82354d60eee3eb5f"]},{"key":"dc:title","label":"Title","values":["Speech Foundation Models for Audio Processing"]}]}],"canonical_facts":{"dc:contributor.advisor":["Knill, katherine"],"dc:contributor.sponsor":["Cambridge University Press & Assessment Studentship"],"dc:creator":["Ma, Rao"],"dc:date.issued":["2025-09-02"],"dc:description.abstract":["Foundation models are trained on massive amounts of data and can be adapted to support a wide range of downstream tasks. Their emergence has significantly shaped the development of natural language and speech technologies. Foundation speech models such as Whisper have shown strong performance across a variety of audio processing tasks, including automatic speech recognition (ASR) and speech translation. Unlike traditional systems that require task-specific architectures and extensive supervision, these models offer a unified and flexible framework that supports generalisation to new domains, languages, and task formulations. In this thesis, we present a comprehensive investigation into the capability, application, and vulnerability of foundation speech models in the context of audio processing, aiming to understand both their potential and limitations. This thesis begins by addressing ASR error correction, a task that aims to refine ASR outputs to improve the accuracy and readability of transcriptions. Traditional approaches typically take the 1-best ASR hypothesis as input, which restricts correction models to the limited context of a single transcription. In this work, we propose methods that leverage the ASR N-best hypotheses, allowing the error correction model to access a richer set of alternatives and make more informed predictions. To further guide the generation process, we introduce various constrained decoding strategies based on either the N-best list or an ASR lattice, enabling the error correction model to better utilise knowledge from the ASR system. Beyond supervised ASR error correction, we also investigate a training-free, zero-shot approach using large language models (LLMs). This method enables plug-and-play integration with existing ASR systems, without requiring access to model weights or additional training data. Experimental results across multiple test sets show that the proposed methods yield consistent performance improvements for two different ASR backends. Next, we investigate cross-lingual transfer in speech translation. Whisper is trained on multilingual speech data in a multitask setting, which suggests that utterances from different languages are projected into a shared semantic space by the acoustic encoder, thereby enabling cross-lingual transfer. To examine this property, we analyse encoder representations through a speech-to-speech retrieval task. Leveraging this shared representation, we extend Whisper’s speech translation capabilities to new source-target language pairs. By modifying the control tokens provided to the decoder, we enable zero-shot speech translation for language pairs not observed during Whisper’s pre-training. Fine-tuning Whisper on English-Chinese translation data further improves performance not only for English inputs but also for other languages, surpassing the zero-shot baseline. Finally, experiments on low-resource languages show that Whisper can translate speech from languages absent during pre-training, illustrating its strong generalisation ability through cross-lingual representations. We then focus on the use of Whisper for a more complex spoken language processing task: spoken grammatical error correction (SGEC), which aims to automatically detect and correct grammatical errors in transcribed speech. Traditional approaches typically rely on cascaded pipelines consisting of separate modules for ASR, disfluency detection, and grammatical error correction. However, such pipelines are prone to information loss and error propagation. To address these limitations, we propose an end-to-end framework that directly generates grammatically corrected text from speech input. To overcome the lack of training data, we develop a large-scale pseudo-labelling strategy that expands the corpus size substantially. Additional gains are achieved through contextual prompt-based learning and a novel alignment-based method that modifies training references to improve feedback precision. The proposed approach outperforms cascaded baseline systems and offers a practical solution for learner feedback applications. Foundation text and vision models have demonstrated emergent abilities to perform tasks beyond their pre-training objectives, a property that makes them especially valuable for real-world applications. In this thesis, we investigate Whisper’s emergent capability for zero-shot audio classification, a task outside its original training objective. We introduce a template-based prompting method that uses decoder probabilities for classification without requiring fine-tuning or labelled data. A simple unsupervised debiasing strategy further improves performance by reweighting class probabilities. The proposed method achieves strong results across eight benchmark datasets and surpasses previous zero-shot baselines. We further show that performance increases with model size, implying that as ASR foundation models scale up, they may exhibit improved zero-shot performance, akin to text-based foundation models. Finally, we address the potential vulnerability of foundation speech models to adversarial manipulation. We investigate universal acoustic adversarial attacks on speech LLMs, where a fixed adversarial audio segment is prepended to the input. This segment is learned using gradient-based optimisation on a small dataset and, once generated, can be reused for any speech input to that model without needing further access to the model or its prompt. In addition to general attacks, we introduce selective attacks that activate only under specific input conditions, such as a specific speaker gender or spoken language. These targeted attacks are more difficult to detect and raise serious concerns regarding fairness and safety. Experimental results show that the proposed attacks are highly effective against versatile speech LLMs such as Qwen2-Audio, highlighting the need for more robust training and defence mechanisms."],"dc:format.checksum.md5":["bef526f17496b2abea977b3ecbe02a73","87eda9de84448d1f82354d60eee3eb5f"],"dc:identifier.doi":["https://doi.org/10.17863/CAM.132340"],"dc:identifier.uri":["https://www.repository.cam.ac.uk/bitstreams/2224e9e6-280f-4c0c-b69b-7d46b9bd1d0b/download"],"dc:language":["eng"],"dc:publisher.institution":["University of Cambridge"],"dc:relation.isreferencedby.uri":["https://www.repository.cam.ac.uk/handle/1810/406210"],"dc:rights":["https://www.repository.cam.ac.uk/bitstreams/652a9f72-2d16-4812-90f1-492b87c83b1f/download","https://creativecommons.org/licenses/by/4.0/"],"dc:subject":["audio processing","speech foundation models","automatic speech recognition","natural language processing","large language models"],"dc:title":["Speech Foundation Models for Audio Processing"],"dc:type":["Thesis"],"dc:type.qualificationlevel":["Doctoral"],"dc:type.qualificationname":["Doctor of Philosophy (PhD)"]},"updated_at":"2026-07-24T01:33:18Z"}