Learning Generalizable Systems by Learning Composable Energy Landscapes

How can we construct intelligent embodied agents in the physical world? Such agents should be able to autonomously solve tasks that have not been seen before, subject to external disturbances in the environment, as well as new combinations of factors such as lighting, varying sensor inputs, and unexpected interactions with agents and other objects. An important subgoal towards constructing such intelligent agents is to construct models that can robustly generalize, not only to distributions of tasks similar to ones seen at training time but also to new unseen distributions. This departs from standard machine learning techniques which usually assume identical training and test distributions. Towards this goal, in this dissertation, we’ll illustrate how we can achieve certain forms of generalization by estimating energy landscapes over possible predictions for each task, with accurate predictions assigned lower energy. This modeling choice formulates prediction as a search process on the energy landscape, enabling zero-shot generalization to new constraints by adapting the energy landscape. In addition, this allows us to generalize to entirely new distributions of tasks in a zero-shot manner by composing multiple learned energy landscapes together. In this dissertation, we first introduce a set of techniques to train energy landscapes and an algebra in which we can compose and discover composable energy landscapes. Next, we illustrate how energy landscapes can be composed in a diverse set of ways, ranging from logical operators, probability distributions, graphical models, constraints, and hierarchical compositions, enabling effective generalization across vision, decision-making, multimodal, and scientific settings.

Name thesis:degree_name

Doctoral

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2025

Author dc:creator

• Du, Yilun

Advisors dc:contributor.advisor

• Kaelbling, Leslie
• Lozano-Pérez, Tomás
• Tenenbaum, Joshua B.

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright retained by author(s)

Licence dc:rights.uri

• https://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/158938

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/158938