Reinforcement Learning for Multi-modal Human-Robot Interaction

This dissertation addresses the challenges of enabling natural and effective collaboration between humans and assistive robots in domestic settings. To support users in activities of daily living (ADLs), assistive robots must understand multimodal inputs, engage in dynamic interaction, and adapt to vague or evolving instructions. We approach this problem through a unified framework that spans low-level interaction policy learning, user simulation, and high-level task planning. First, we propose a neural network-based multimodal user simulator trained on real-world demonstrations from the ELDERLY-AT-HOME corpus. The simulator generates realistic human behavior across speech, gestures, and haptic actions, enabling scalable reinforcement learning (RL) for collaborative tasks. We then develop an interpretable RL-based Interaction Manager that learns to take the role of a helper (HEL) in a collaborative object-finding task, demonstrating high accuracy and user satisfaction through a comprehensive user study. Finally, we introduce IteraPlan, an iterative task-level planning framework that uses Large Language Models (LLMs) to generate and refine plans from vague natural language commands. Unlike existing systems that rely on structured prompts and fixed plans, IteraPlan integrates real-time human feedback to adapt its behavior on the fly.

Author dc:creator

• Afagh Mehri Shervedani (23291314)

• An End-to-End Human Simulator for Task-Oriented Multimodal Human-Robot Collaboration
• Multimodal Reinforcement Learning for Robots Collaborating with Humans
• From Vague Instructions to Task Plans: A Feedback-Driven HRC Task Planning Framework based on LLMs

Statement dc:rights

• In Copyright

DOI dc:identifier

https://doi.org/10.25417/uic.31451053.v1

OAI identifier oai:identifier

oai:figshare.com:article/31451053