Understanding sketch-and-speech descriptions of machines

In this thesis, we present PhysInk, a sketching application that provides a natural way to describe physical structure and behavior to a computer. The user describes structure by sketching physical objects and constraints on a canvas backed by a 2D physics engine. The user can then move the objects to explicitly demonstrate physical behavior and can utter simple speech commands to label objects and events. These interactions are captured in PhysInk's timeline - a causal graph of physical events - which is used to build a deeper understanding of behavior. The events capture the geometry of movement and contact between objects, while the timeline captures the causal relationships between events. Capturing behavior in this framework enables a more intelligent conversation with the user about function. First, it enables assisted editing, where PhysInk propagates the user's edits of behavior through the timeline in order to maintain a logical sequence of events at all times. Second, it allows users to produce a physically-realistic simulation of the behavior that they have described. These features demonstrate the usefulness and depth that the timeline offers as a knowledge representation for behavior, making PhysInk the first in a new class of design tools that focuses on function, as well as form.

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2012

Author dc:creator

• Scott, Jeremy (Jeremy Kenneth)

Advisor dc:contributor.advisor

• Randall Davis.

• Electrical Engineering and Computer Science.

Statement dc:rights

• M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

http://hdl.handle.net/1721.1/78470

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/78470