Towards abstractive captioning of infographics

Machine understanding of text-based narratives have predominantly focused on documents with rigid hierarchical structures and sequentially ordered inputs. These inputs include documents such as news stories, encyclopedia entries, books, and many others. However, little research has focused on understanding text-based information without this structure. Current text understanding models fail when information is presented in less structured ways, without a clear and pre-defined spatial arrangement of the content. This thesis explores a subset of components required for understanding infographics -- documents whose structure is not necessarily linear and whose content may involve a variety of images. We expand on state-of-the-art methodologies in character recognition and text summarization in order to better understand how to process content without a pre-determined spatial arrangement, and subsequently generate captions for given infographics automatically. To shine light at the reasoning behind the captions being generated, we develop a graphical user interface that helps visualize the portions of a document being used when generating specific parts of a caption.

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2018

Author dc:creator

• Landman, Nathan, M. Eng. Massachusetts Institute of Technology

Advisor dc:contributor.advisor

• Frédo Durand

• Electrical Engineering and Computer Science.

Statement dc:rights

• MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

http://hdl.handle.net/1721.1/119743

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/119743