Word confidence measures for machine translation

Due to continuous research which led to improved concepts and algorithms, the quality of automatically generated translation has significantly improved in recent years. However, the performance of machine translation systems is still not perfect. For human users dealing with these systems, it is desirable to obtain a reliable indication of possible errors in the system output. The same holds for applications based on machine translation technologies. They could explore the knowledge about possible mistakes.The aim of this work is to provide knowledge about when a translation generated by the system is incorrect by calculating measures of confidence for each word in this translation. This topic has hardly been investigated in machine translation before. Different ways of determining confidence measures are proposed and experimentally evaluated in this thesis. The basic concept behind all these approaches are word posterior probabilities.The main problem which has to be solved for the computation of word posterior probabilities is to define the underlying concept. There exists no intuitive definition of this concept. Possible approaches include the word posterior probability of a word based on its position in the sentence and the occurrence in any position. Several solutions to this problem are presented in this thesis. Furthermore, different approaches to the calculation of word posterior probabilities are introduced and compared. They can be divided into two categories: system-based methods which explore knowledge provided by the translation system that has generated the translations, and direct methods which are independent of the translation system. The system-based techniques make use of system output, such as word graphs or N-best lists. The direct confidence measures take other knowledge sources, such as word or phrase lexica, into account. The word posterior probabilities can directly be applied as confidence measuresas follows: For a given translation generated by a machine translation system, the posterior probabilities of all words are determined and compared to a threshold. All words whose posterior probability is above this threshold are tagged as correct and all others are tagged as incorrect. To evaluate the proposed confidence measures, the information on which words are correct is needed. In machine translation, it is not intuitively clear how to determine thecorrectness of single words. As a solution to this problem, several different ways of deriving word error measures from existing machine translation evaluation metrics are investigated. From the formulation of the posterior risk for different error measures, a theoretical foundation of the word posterior probabilities is derived.The different confidence measures explore information from various knowledge sources, such as sentence probabilities provided by the machine translationsystem and statistical word and phrase lexica. To explore the knowledge fromall these sources, a combination of several confidence measures is investigated. The suggested methods are evaluated on different translation tasks and language pairs. In order to assess the general discriminative power of the confidence measures, they are tested on output from four different machine translation systems. Three of those are state-of-the-art phrase-based systems, and the fourth is an established rule-based system. A significant improvement in terms of confidence error rate is achieved in all settings.In this work, applications of confidence measures that improve translation quality of state-of-the-art systems are investigated. These include rescoringwith confidence measures and their use in an interactive machine translation system.

Grantor dc:publisher

Publikationsserver der RWTH Aachen University

Year dc:date

2006

Author dc:creator

• Ueffing, Nicola

Contributors dc:contributor

• Ney, Hermann

• info:eu-repo/classification/ddc/004
• Informatik
• Sprachverarbeitung
• Automatische Übersetzung
• Konfidenzmaße
• statistische Übersetzung
• Wortposteriorwahrscheinlichkeiten
• statistical machine translation
• confidence measures
• confidence estimation
• word posterior probabilities

Statement dc:rights

• info:eu-repo/semantics/openAccess

Language dc:language

eng

Identifier

https://publications.rwth-aachen.de/search?p=id:%22RWTH-CONV-121561%22

OAI identifier oai:identifier

oai:publications.rwth-aachen.de:59808