Learning structured models for robotic manipulation of deformable objects

Robotic manipulation of deformable objects poses significant challenges due to their complex, nonlinear behaviors, high degrees of freedom, and variable material properties. Traditional model-based techniques are often inadequate at representing these subtleties in the real world, while many learning-based strategies struggle to generalize across different object shapes, poses, and environmental conditions. This thesis presents two novel and unified frameworks that address these challenges through the structured and adaptive modeling of diverse deformable objects. The first framework, Material-Adaptive Graph-Based Neural Dynamics (AdaptiGraph), uses particle-based representations enriched with material type information and continuous physical property parameters. By incorporating these features into a graph neural network and applying a test-time few-shot adaptation strategy, AdaptiGraph delivers precise dynamics predictions and facilitates online estimation of physical properties across various rigid and soft materials. Second, a Particle-Grid Neural Dynamics framework is proposed to directly learn the dynamics of deformable objects from depth images. This approach leverages the innate geometric and physical structure of particle-grid representations to capture fine-scale features, thereby providing accurate predictions of how deformable objects evolve during robotic interactions while remaining robust even under partial observation conditions. Extensive experiments in real-world robotics settings have demonstrated that these frameworks considerably improve manipulation performance in tasks involving 1D, 2D, and 3D deformable objects, including rope straightening, granular material gathering, cloth manipulation, and plush toy relocation. Together, these contributions advance the development of data-efficient robotic systems capable of operating in unstructured environments with diverse deformable materials, while also opening avenues for further research in adaptive control and deformable object simulation.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Li, Baoyu

Contributors dc:contributor

• Hauser, Kris

• Robot Learning
• Robotic Manipulation
• Deformable Object Manipulation
• Graph-Based Neural Dynamics

Statement dc:rights

• Copyright 2025 Baoyu Li

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129326

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129326