Entity-based scene understanding

Unifying multiple descriptions to determine the details of an everyday event can be a challenging task for humans. Though incorporating other modalities like images or videos can help humans unify such descriptions, this remains a challenging task for computational systems. We define entity-based scene understanding as the task of identifying the entities in a visual scene from multiple descriptions. This task subsumes coreference resolution, bridging resolution, and grounding to produce mutually consistent relations between entity mentions and groundings between mentions and image regions. Using neural classifiers and integer linear program inference, we show that grounding is improved when forced to conform to relation predictions. We introduce the Flickr30k Entities v2 dataset, and show how our methods can be used to automatically generate similarly rich annotations for the MSCOCO dataset.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2018

Author dc:creator

• Cervantes, Christopher Michael

Contributors dc:contributor

• Hockenmaier, Julia

• coreference, bridging, grounding, neural networks, LSTM, Flickr30k Entities, MSCOCO

Statement dc:rights

• Copyright 2018 Christopher Cervantes

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/101072

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/101072