A uniform representation for visual concepts

We present a method for learning visually-grounded word meanings, given as input a set of videos paired with natural-language sentences describing them. Our method uses a uniform feature representation for all words and word types rather than relying on handcrafted features specific to each word. We learn words in a weakly-supervised manner, with no need for annotated bounding boxes around objects of interest. We encode words as Hidden Markov models such that word models can be composed according to a sentence's semantic structure to efficiently recognize events in videos. We use a discriminative variant of Baum-Welch to learn the parameters for our word models, and demonstrate that our approach is able to learn words capturing appearance, spatial relations, and temporal dynamics.

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2016

Author dc:creator

• Rakover, Nicolas

Advisor dc:contributor.advisor

• Boris Katz.

• Electrical Engineering and Computer Science.

Statement dc:rights

• M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

http://hdl.handle.net/1721.1/105965

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/105965