Automated Multiple-Choice Question Generation and Analysis for Language Learning Assessment

This thesis investigates the application of natural language processing (NLP) techniques to the development and evaluation of language assessment tasks in computer-assisted language learning. It focuses on three interrelated areas: evaluating candidate responses, analysing multiple-choice reading comprehension items, and generating assessment questions. The work is situated within the broader aim of improving the efficiency and reliability of standardised assessments through automation. The first area of study examines the detection of off-topic responses in spoken language assessments. This is treated as a form of unanswerability detection: just as multiple-choice question generation must guard against producing items with no valid answer, spoken tasks can generate responses that fail to address the prompt. Identifying such cases is an important step in ensuring the validity of assessment items, and the methods and insights developed here carry forward to the broader pipeline of automated multiple-choice question generation and analysis. The section considers the impact of automatic speech recognition errors and grammatical variation among non-native speakers, while proposing the use of synthetic training data to address the scarcity of annotated examples. A normalised evaluation metric is also introduced to enable fairer comparisons across datasets. The second part of the thesis turns to multiple-choice question answering (MCQA) as a means of supporting reading comprehension assessment. Rather than focusing solely on answer accuracy, the work analyses the distribution of responses predicted by automated systems, showing how this distribution can be used to simulate candidate behaviour and support item selection during pre-testing. Additional attention is given to the detection of unanswerable questions and to quantifying model uncertainty, both of which inform the reliability of MCQA systems in assessment contexts. Building on this, the third area explores automated methods for analysing multiple-choice questions. The thesis investigates how individual components of a question — such as the context passage or prompt — shape response patterns, and to what extent questions genuinely require reading comprehension. Computational methods for estimating relative question difficulty are also examined, offering support for more consistent and scalable pre-testing practices. Finally, the thesis addresses the generation of multiple-choice questions, integrating recent advances in instruction-tuned language models. It discusses how source texts can be adapted, for instance through readability modification, to better support question generation for particular proficiency levels. The quality of the resulting items is assessed using automated tools, with a view to filtering suitable questions for further consideration. By situating question generation within the broader assessment pipeline, the work contributes to a more streamlined process from item creation to pre-testing. Taken together, the contributions of this thesis reflect a structured investigation into how NLP methods can support the design, analysis, and evaluation of language assessments, with the aim of making them more systematic and adaptable.

Name dc:type.qualificationname

Doctor of Philosophy (PhD)

Level dc:type.qualificationlevel

Doctoral

Grantor dc:publisher.institution

University of Cambridge

Year dc:date.issued

2025

Author dc:creator

• Raina, Vatsal

Advisor dc:contributor.advisor

• Gales, Mark

• NLP
• Language Assessment
• Question Generation
• Question Answering

Licence

• https://www.repository.cam.ac.uk/bitstreams/f3582ff0-1d8a-4744-a2d2-3118b7c31f92/download
• http://purl.org/NET/rdflicense/allrightsreserved

Language dc:language

eng

Author Identifier

0000-0002-3422-6513

OAI identifier oai:identifier

oai:www.repository.cam.ac.uk:1810/392519