Compositional Robot Learning for Generalizable Interactions

To understand environments effectively and to interact safely with humans, robots must generalize their learned models to scenarios they have never been trained on before, such as new commands and new agents. Humans have shown a remarkable ability to compose concepts they have learned before in order to interpret and to act in a novel environment. In contrast, many deep-learning based methods fail at compositional generalization, i.e., an ability to generalize to novel combinations of concepts that have not been seen before in training. This thesis presents several learning-based approaches that leverage compositionally to enable generalization in various reasoning skills such as language understanding and social interactions. First, we show how we can derive networks directly from compositional linguistic structures to enable robots to follow novel commands and act rationally in new scenarios. These networks are extended to encode temporal relationships in linear temporal logic and augmented with sampling-based planners to plan in continuous space. Then we show how we can formulate social interactions as reward operations and apply recursive reward estimation to enable robots to reason about novel social interactions. Finally, we explore an alternative approach to incorporate compositionality when there is no explicit compositional structure — using language as an intermediate representation for internal reasoning. We show how this linguistic representation can help robots predict the trajectories of other agents.

Name thesis:degree_name

Doctoral

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2022

Author dc:creator

• Kuo, Yen-Ling

Advisors dc:contributor.advisor

• Katz, Boris
• Barbu, Andrei

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright MIT

Licence dc:rights.uri

• http://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/144669

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/144669