Fine-grained error analysis in machine translation

This dissertation focuses on the fine-grained evaluation method for machine translation. We define fine-grained tests as methods which evaluate translation quality with respect to a particular phenomenon. These methods contrast with one-dimensional evaluation methods, which quantify translation quality as a single number. The BLEU score, and more recent neural translation metrics like XCOMET, are examples of such metrics. In this work we demonstrate, from our own experiments and from a review of related work, that fine-grained evaluation methods are more useful tools for translation evaluation. We argue that this is the case because translation quality is inherently multidimensional. A correct translation must not only convey the meaning of the source language, but must also be grammatical in the target language. These two factors themselves are complex phenomena, with many orthogonal components. Fine-grained tests are thus well-suited to the fundamentally multidimensional nature of translation quality. In our first experiment, we demonstrate that fine-grained tests are uniquely useful for understanding how the training neural machine translation system unfolds. Additionally, through a preliminary experiment targeting the translation of the Turkish evidentiality morpheme, we observe translation system behaviors that suggest they may not have acquired evidentiality. These findings, which we were not able to explain with linguistic or sentence complexity features, motivate further research on this phenomenon. We present this work as a review of related work on fine-grained error analysis in machine translation, supplemented with our own findings, analysis, and recommendations. We do this to aid the future development of such tests, which we believe will be crucial for navigating our current era which is increasingly dominated by large language models.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Linguistics

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Dell, Brennan

Contributors dc:contributor

• Dunn, Jonathan
• Yoon, James
• Maskharashvili, Aleksandre
• Tang, Yan

• Machine Translation
• Computational Linguistics
• Turkish
• Finnish
• Evidentiality

Statement dc:rights

• Copyright 2025 Brennan Dell

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129863

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129863