VirtualHome: Building Socially Intelligent Agents via Simulation

While remarkable progress has been made in building autonomous agents that can help us at complex tasks, these have typically been studied in isolated environments. To build agents that can be deployed in the real world, we need to study them in more human-centric settings, where they can interact with other humans and agents. Moreover, for agents to assist us effectively, they need to be able to operate in these environments while understanding our intentions and beliefs, and learn to coordinate with us effectively and safely. This thesis investigates the development of such assistive agents through simulation environments. In the first part of the thesis, we introduce VirtualHome, a multi-agent platform for simulating human activities in household environments, and introduce a knowledge base of daily human activities that can be executed in the simulator. Then, we present agents that can perform different tasks in the environment given human descriptions or demonstrations of the activity. Finally, we study agents that can perform activities together with other humans in the environment. We propose a framework to simulate humans in the environment at scale and propose two challenges, Watch-And-Help and Online-Watch-And-Help, to benchmark the performance of different assistive agents, and test their effectiveness in assisting real humans performing activities in VirtualHome. Together, the methods and tools presented in this thesis provide a way to study assistive agents in simulation, allowing to develop these agents safely and at scale before deploying them into the real world.

Name thesis:degree_name

Doctoral

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2023

Author dc:creator

• Puig Fernández, Xavier

Advisor dc:contributor.advisor

• Torralba, Antonio

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright MIT

Licence dc:rights.uri

• http://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/150100

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/150100