{"id":{"repo_id":"uic","oai_identifier":"oai:figshare.com:article/31451053"},"canonical_url":"https://search.dev.ndltd.org/etd/uic/oai:figshare.com:article/31451053","repository":{"repo_id":"uic","name":"University of Illinois - Chicago","base_url":"https://api.figshare.com/v2/oai"},"display":{"title":"Reinforcement Learning for Multi-modal Human-Robot Interaction","abstract":"This dissertation addresses the challenges of enabling natural and effective collaboration between humans and assistive robots in domestic settings. To support users in activities of daily living (ADLs), assistive robots must understand multimodal inputs, engage in dynamic interaction, and adapt to vague or evolving instructions. We approach this problem through a unified framework that spans low-level interaction policy learning, user simulation, and high-level task planning. First, we propose a neural network-based multimodal user simulator trained on real-world demonstrations from the ELDERLY-AT-HOME corpus. The simulator generates realistic human behavior across speech, gestures, and haptic actions, enabling scalable reinforcement learning (RL) for collaborative tasks. We then develop an interpretable RL-based Interaction Manager that learns to take the role of a helper (HEL) in a collaborative object-finding task, demonstrating high accuracy and user satisfaction through a comprehensive user study. Finally, we introduce IteraPlan, an iterative task-level planning framework that uses Large Language Models (LLMs) to generate and refine plans from vague natural language commands. Unlike existing systems that rely on structured prompts and fixed plans, IteraPlan integrates real-time human feedback to adapt its behavior on the fly.","abstract_html":"This dissertation addresses the challenges of enabling natural and effective collaboration between humans and assistive robots in domestic settings. To support users in activities of daily living (ADLs), assistive robots must understand multimodal inputs, engage in dynamic interaction, and adapt to vague or evolving instructions. We approach this problem through a unified framework that spans low-level interaction policy learning, user simulation, and high-level task planning. First, we propose a neural network-based multimodal user simulator trained on real-world demonstrations from the ELDERLY-AT-HOME corpus. The simulator generates realistic human behavior across speech, gestures, and haptic actions, enabling scalable reinforcement learning (RL) for collaborative tasks. We then develop an interpretable RL-based Interaction Manager that learns to take the role of a helper (HEL) in a collaborative object-finding task, demonstrating high accuracy and user satisfaction through a comprehensive user study. Finally, we introduce IteraPlan, an iterative task-level planning framework that uses Large Language Models (LLMs) to generate and refine plans from vague natural language commands. Unlike existing systems that rely on structured prompts and fixed plans, IteraPlan integrates real-time human feedback to adapt its behavior on the fly.","abstract_has_math":false,"creators":["Afagh Mehri Shervedani (23291314)"],"institution":null,"degree_name":null,"degree_level":null,"degree_discipline":null,"degree_department":null,"school":null,"contributors":[],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025-12-01T00:00:00Z","date_published":"2025-12-01T00:00:00Z","updated_at":"2026-07-27T21:34:19Z","subjects":["An End-to-End Human Simulator for Task-Oriented Multimodal Human-Robot Collaboration","Multimodal Reinforcement Learning for Robots Collaborating with Humans","From Vague Instructions to Task Plans: A Feedback-Driven HRC Task Planning Framework based on LLMs"],"languages":[],"rights":["In Copyright"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://doi.org/10.25417/uic.31451053.v1","outbound_label":"DOI","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:creator","label":"Author","values":["Afagh Mehri Shervedani (23291314)"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2025-12-01T00:00:00Z"]},{"key":"dc:relation","label":"Dc Relation","values":["https://figshare.com/articles/thesis/Reinforcement_Learning_for_Multi-modal_Human-Robot_Interaction/31451053"]},{"key":"dc:type","label":"Dc Type","values":["Text","Thesis"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["An End-to-End Human Simulator for Task-Oriented Multimodal Human-Robot Collaboration","Multimodal Reinforcement Learning for Robots Collaborating with Humans","From Vague Instructions to Task Plans: A Feedback-Driven HRC Task Planning Framework based on LLMs"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:rights","label":"Dc Rights","values":["In Copyright"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["10.25417/uic.31451053.v1"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["This dissertation addresses the challenges of enabling natural and effective collaboration between humans and assistive robots in domestic settings. To support users in activities of daily living (ADLs), assistive robots must understand multimodal inputs, engage in dynamic interaction, and adapt to vague or evolving instructions. We approach this problem through a unified framework that spans low-level interaction policy learning, user simulation, and high-level task planning. First, we propose a neural network-based multimodal user simulator trained on real-world demonstrations from the ELDERLY-AT-HOME corpus. The simulator generates realistic human behavior across speech, gestures, and haptic actions, enabling scalable reinforcement learning (RL) for collaborative tasks. We then develop an interpretable RL-based Interaction Manager that learns to take the role of a helper (HEL) in a collaborative object-finding task, demonstrating high accuracy and user satisfaction through a comprehensive user study. Finally, we introduce IteraPlan, an iterative task-level planning framework that uses Large Language Models (LLMs) to generate and refine plans from vague natural language commands. Unlike existing systems that rely on structured prompts and fixed plans, IteraPlan integrates real-time human feedback to adapt its behavior on the fly."]},{"key":"dc:title","label":"Title","values":["Reinforcement Learning for Multi-modal Human-Robot Interaction"]}]}],"canonical_facts":{"dc:creator":["Afagh Mehri Shervedani (23291314)"],"dc:date":["2025-12-01T00:00:00Z"],"dc:description":["This dissertation addresses the challenges of enabling natural and effective collaboration between humans and assistive robots in domestic settings. To support users in activities of daily living (ADLs), assistive robots must understand multimodal inputs, engage in dynamic interaction, and adapt to vague or evolving instructions. We approach this problem through a unified framework that spans low-level interaction policy learning, user simulation, and high-level task planning. First, we propose a neural network-based multimodal user simulator trained on real-world demonstrations from the ELDERLY-AT-HOME corpus. The simulator generates realistic human behavior across speech, gestures, and haptic actions, enabling scalable reinforcement learning (RL) for collaborative tasks. We then develop an interpretable RL-based Interaction Manager that learns to take the role of a helper (HEL) in a collaborative object-finding task, demonstrating high accuracy and user satisfaction through a comprehensive user study. Finally, we introduce IteraPlan, an iterative task-level planning framework that uses Large Language Models (LLMs) to generate and refine plans from vague natural language commands. Unlike existing systems that rely on structured prompts and fixed plans, IteraPlan integrates real-time human feedback to adapt its behavior on the fly."],"dc:identifier":["10.25417/uic.31451053.v1"],"dc:relation":["https://figshare.com/articles/thesis/Reinforcement_Learning_for_Multi-modal_Human-Robot_Interaction/31451053"],"dc:rights":["In Copyright"],"dc:subject":["An End-to-End Human Simulator for Task-Oriented Multimodal Human-Robot Collaboration","Multimodal Reinforcement Learning for Robots Collaborating with Humans","From Vague Instructions to Task Plans: A Feedback-Driven HRC Task Planning Framework based on LLMs"],"dc:title":["Reinforcement Learning for Multi-modal Human-Robot Interaction"],"dc:type":["Text","Thesis"]},"updated_at":"2026-07-27T21:34:19Z"}