Collaborative embodied agents

In recent years, research in computer vision has increasingly focused on solving realistic tasks in an interactive or embodied setting. Research in embodied agents lies at the intersection of computer vision, reinforcement learning, and language understanding. Prior research and tasks (such as navigation, question answering, language grounded navigation) have equipping agents with intelligent skills with a focus on a single agent. Going forward, we believe multi-agent learning can facilitate solving increasingly complex tasks such as driving, playing sports, or moving heavy objects, or rearranging them inside the house. Taking first steps in this direction, we build AI Agents that can collaborate and communicate in virtual visual worlds. Particularly, we'll discuss collaborative learning within both homogeneous and heterogeneous sets of agents. For collaborative homogeneous agents, we include (1) formulation of collaborative tasks and effect of 'explicit' and 'implicit' communication and (2) rich 'mixture-of-marginals' policies that overcome restrictions of existing decentralized multi-agent policies. For collaborative heterogeneous agents, we include research that (3) allow learning of embodied agents from free supervision from simplistic gridworlds via a 'GridToPix' methodology and (4) how to collaborate and learn from teachers that enjoy more privilege than the student. Overall, this dissertation lays the foundations of how visual AI agents can develop skills outside the silo of learning by themselves i.e. social learning. Going forward, it would be exciting to see how to learn from other agents in your surroundings from in-the-wild videos and test sim-to-real transfer of embodied ideas and results to robots.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2022

Author dc:creator

• Jain, Unnat

Contributors dc:contributor

• Schwing, Alexander
• Lazebnik, Svetlana
• Hoiem, Derek
• Jiang, Nan
• Grauman, Kristen

• embodied agents
• visual navigation
• multi-agent systems
• multi-agent RL
• imitation learning
• simulation
• virtual environments
• embodied AI

Statement dc:rights

• Copyright 2022 Unnat Jain

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/115410

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F115410