Visually grounded virtual accelerometers : a longitudinal video investigation of dyadic bodily dynamics around the time of word acquisition

Human movement encodes information about internal states and goals. When these goals involve dyadic interactions, such as in language acquisition, the nature of the movement and proximity become representative, allowing parts of our internal states to manifest. We propose an approach called Visually Grounded Virtual Accelerometers (VGVA), to aid with ecologically-valid video analysis investigations, involving humans during dyadic interactions. Utilizing the Human Speechome (HSP) [1] video corpus database, we examine a dyadic interaction paradigm taken from the caregiver-child ecology, during language acquisition. We proceed to characterize human interaction in a video cross-modally; by visually detecting and assessing the child's bodily dynamics in a video, grounded on the caregiver's bodily dynamics of the same video and the related HSP speech transcriptions [2]. Potential applications include analyzing a child's language acquisition, establishing longitudinal diagnostic means for child developmental disorders and generally establishing a metric of effective human communication on dyadic interactions under a video surveillance system. In this thesis, we examine word-learning transcribed video episodes before and after the age of the word's acquisition (AOA). As auditory stimulus is uttered from the caregiver, points along the VGVA tracked sequences corresponding to the onset and post-onset of the child-caregiver bodily responses, are used to longitudinally mark and characterize episodes of word learning. We report a systematic shift in terms of caregiver-child synchrony in motion and turning behavior, tied to exposures of the target word around the time the child begins to understand and thus respond to instances of the spoken word. The systematic shift, diminishes gradually after the age of word acquisition (AOA).

Department dc:contributor.department

Massachusetts Institute of Technology. Dept. of Architecture. Program in Media Arts and Sciences.

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2010

Author dc:creator

• Tsourides, Kleovoulos (Kleovoulos Leo)

Advisor dc:contributor.advisor

• Deb Roy.

• Architecture. Program in Media Arts and Sciences.

Statement dc:rights

• M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

http://hdl.handle.net/1721.1/65179

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/65179