Learning 3D Modeling and Simulation From and For the Real World

Humans have extraordinary capabilities of comprehending and reasoning about our 3D visual world. With just a few casual glances, we can grasp the 3D structure and appearance of our surroundings and imagine all sorts of “what-if” scenarios in our minds. Existing 3D systems, in contrast, cannot. They lack structural understanding of the world and often break apart when moved to unconstrained, partially-observed, and noisy environments. To alleviate the challenge, this thesis focus on developing robust computational tools that can effectively perceive, model, and simulate the 3D world from unconstrained sensory data. We investigate the full spectrum of dynamic 3D world understanding: from robot localization to recognition, from static 3D reconstruction to dynamic motion estimation, and from closed-loop simulation to 3D generation. By examining these tasks not only in controlled settings, but also in sparse, noisy, and sometimes even extreme real-world settings, we aim to answer the following two questions: (i) how to robustly model and reason about the visible world that we see; and (ii) how to hallucinate the unseen and imagine novel scenarios in a realistic fashion.

Name thesis:degree_name

Doctoral

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2024

Author dc:creator

• Ma, Wei-Chiu

Advisor dc:contributor.advisor

• Torralba, Antonio

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright retained by author(s)

Licence dc:rights.uri

• https://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/153905

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/153905