Structured model learning for adaptive robot generalists

As robots move from factories into homes, hospitals, and warehouses, the grand challenge is to create adaptive robot generalists: systems that can learn diverse manipulation skills, transfer knowledge across tasks and embodiments, and operate robustly under varying sensing conditions. However, robotic manipulation in such unstructured environments requires integrating diverse sensory modalities, long-horizon reasoning, and learning from heterogeneous data sources. Traditional monolithic learning approaches remain brittle, lack modularity, and struggle to generalize across tasks, embodiments, and sensing configurations. This dissertation presents a unified framework based on structured model learning that decomposes the robot learning problem into modular, reusable components. This approach yields policies that generalize across tasks, adapt to new contexts, and operate robustly under diverse conditions. The framework is developed progressively across manipulation challenges of increasing complexity. It begins by integrating learned policies with analytical models for hybrid control in traffic simulation and robotic manipulation. This foundation extends to hierarchical decomposition, where tasks are separated into high-level goals and low-level skills learned from direct physical human guidance. For contact-rich scenarios involving multiple objects, learned dynamics models combined with behavior primitives enable reasoning about object interactions. The framework further separates state prediction (using diffusion models) from action generation (using inverse dynamics models), enabling coordinated bimanual manipulation of deformable objects. Beyond structural decomposition within individual tasks, the framework enables generalization across embodiments and sensing modalities. Tool-centric representations bridge embodiment differences, enabling robots to learn manipulation skills from human videos. For multimodal sensor fusion, the framework uses separate policy experts for vision and touch, coordinated through learned routing that dynamically balances their contributions based on task context. Validated on tasks including contact-rich stowing, occluded object retrieval, in-hand reorientation, and coordinated bimanual manipulation, the results demonstrate that structured model learning provides a scalable foundation for adaptive robot generalists capable of operating effectively in complex real-world environments.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Chen, Haonan

Contributors dc:contributor

• Driggs-Campbell, Katherine
• Li, Yunzhu
• Amato, Nancy
• Schwing, Alexander

• Robotics
• Machine Learning
• Artificial Intelligence

Statement dc:rights

• Copyright 2025 Haonan Chen

Language dc:language

en

Handle dc:identifier

https://hdl.handle.net/2142/132575

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/132575