Multimodal machine translation

Over the past few years, there has been a lot of progress being made in machine translation through deep learning networks. But there is relatively lesser progress made in using images to catalyze the translation tasks. In this study, we explore various models to incorporate the image features in the machine translation models. We start with a monomodal translation model which uses only textual features. We extend this model to develop the multimodal system which incorporates the visual features related to the source sentence. We also propose a multitask system which uses image captioning task to aid the translation task. Our models are tested on multiple datasets using the automatic evaluation metrics like METEOR and BLEU. The experiments show that the proposed models outperform the text-only baseline model.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2018

Author dc:creator

• Dave, Mihika

Contributors dc:contributor

• Hockenmaier, Julia

• multimodal machine translation
• neural machine translation
• multi-task learning
• image captioning

Statement dc:rights

• Copyright 2018 Mihika Dave

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/101374

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/101374