Data-driven bimanual catching: Integrating model-based and learning-based approaches for dynamic object grasping

This thesis presents a novel approach to dynamic object catching using a bimanual robotic system, which integrates both model-based and data-driven methods. Traditional unimanual robotic manipulation systems have limitations in handling large, complex, fast-moving, or irregularly shaped objects. Catching with two hands is more intuitive to humans as it provides greater stability and balance compared to using one hand. We aim to have the robotic system mimic human motion patterns, making it capable of handling a broader range of tasks. We propose a teacher-student policy training framework consisting of a model-based teacher policy and a learning-based student policy. Both policies take raw camera input and output the target grippers' poses for catching objects in flight. The model-based approach utilizes a sampling based technique to generate optimal grasp candidates, and it serves as both a baseline and a foundation for data collection used by the data-driven system. The data-driven approach incorporates PointNet++ and a Transformer encoder to process spatial-temporal point cloud data. Additionally, the network is combined with a physics-based ballistic motion model to estimate the object’s landing position. The deep learning model predicts the 6D rotation for both grippers and 3D translation residuals compensated for object-specific dynamic not captured by the physics-based model. The proposed data-driven system is evaluated in a simulated environment, and it demonstrates significant improvements in grasp success rates and generalization across various object types compared to the baseline model-based method. The results highlight the potential for deploying such a system in real-world applications.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Xu, Yiyang

Contributors dc:contributor

• Driggs-Campbell, Katherine

• Bimanual Robotic Arms
• Robotic Catching

Statement dc:rights

• Copyright 2024 Yiyang Xu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/127288

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F127288