Bridging prediction and planning for human-centered autonomy

Human-centered autonomy needs to develop a strong understanding of human behavior and use the understanding effectively to accomplish challenging human-involved tasks. This dissertation studies human prediction, robot planning, and integration of both to build human-centered autonomy frameworks, and investigates multiple domains including autonomous driving, social navigation, collaborative manufacturing, and collaborative cooking. This dissertation has three parts of studies which span from human prediction, through integration of prediction and planning, to robot planning across the levels of partially observable intent and fully observable trajectory. In the part of human prediction, I introduce a simultaneous intention estimation and trajectory prediction algorithm by combining particle filtering and recurrent neural networks for modeling pedestrians in autonomous driving applications. I investigate pedestrian attention during crowd motion and observe that an agent pays attention to a limited number of neighbors. I use Gumbel Softmax to learn a sparse interaction graph of agents, which is used to aggregate information from the most important neighbors to make multi-agent trajectory prediction. In the part of integration of prediction and planning, I derive intention tracking on a probabilistic graphical model of intention-evolving human-robot collaboration, and extend the derivation to intention hierarchy. I track human intentions at different levels to switch robot control mode and guide robot task and motion planning in a manufacturing setting. I develop language-driven intention tracking by proposing a language probabilistic graphical model and harnessing large language models and vision language models. I use language-driven intention tracking to build an embodied artificial intelligence agent capable of open world understanding, open human intention tracking, and intention-grounded planning, and demonstrate a robot sous-chef application. In addition, I study how to take human trajectory prediction error into consideration for probabilistically safe crowd navigation. I design interaction-aware conformal prediction algorithm to capture mutual influence between a robot and humans during navigation, by alternating uncertainty-aware model predictive control with calibrated human trajectory prediction error, and decision-dependent conformal prediction with human simulation assuming the robot executes the generated motion plan. In the part of robot planning, I present a motion planning algorithm Neural Informed rapidly-exploring random tree star (NIRRT*) by integrating a point-based neural network for guidance state inference into Informed RRT*. In NIRRT*, the Informed RRT* helps the point-based network focus inference on the region relevant to the planning problem for solution improvement, and the point-based network assists the Informed RRT* to sample critical states in the focused subset for optimal convergence acceleration.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Huang, Zhe

Contributors dc:contributor

• Driggs-Campbell, Katherine
• Gupta, Saurabh
• Amato, Nancy
• Mitra, Sayan

• Robotics
• Artificial Intelligence
• Human-robot Interaction

Statement dc:rights

• Copyright 2024 Zhe Huang

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/127228

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F127228