Building Intelligence that can Interact with the Physical World

Recent advances in Artificial Intelligence (AI) have demonstrated remarkable success in parsing, reasoning, and generating digital content across modalities such as natural language, speech, images, videos, and 3D data. However, these breakthroughs have yet to extend meaningfully beyond the digital realm into the physical world. Developing AI for physical interaction poses challenges such as limited grounding, scarce physical data, and high reliability demands in safety-critical settings. This thesis takes a holistic approach to building intelligence that can interact with the physical world – through the lenses of data, brain, and body. Data is the fuel powering highly capable AI systems. We present methods for data-driven simulation that synthesize sensor measurements from physical processes, and knowledge-driven simulation that leverages large language models to generate actor behaviors and scenarios. By reverse engineering the generative processes behind physical data, we address data scarcity while enabling scalable and effective evaluation. The brain, driven by data, demands a deep understanding of the physical world and reliable interaction with it. We introduce methods to bridge the internet-scale knowledge of digital AI with the physical world to improve generalization and interpretability. For greater reliability, we integrate control-theoretic modules into AI models to enable certifiability. Beyond the behavioral intelligence, the body plays a crucial role in physical interaction. We demonstrate how morphological intelligence can emerge from computation and show how pre-trained generative AI models (brain), when augmented with physics-based simulation that provides feedback on generated data, can be applied to robot design. To this end, this thesis explores how digital AI can be extended into the physical world through a comprehensive investigation of data, brain, and body – laying the groundwork for building physical AI.

Name thesis:degree_name

Doctoral

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2025

Author dc:creator

• Wang, Tsun-Hsuan (Johnson)

Advisor dc:contributor.advisor

• Rus, Daniela L.

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright retained by author(s)

Licence dc:rights.uri

• https://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/164123

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/164123