{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/121258"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/121258","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Reinforcement learning with offline data: foundations and algorithms","abstract":"Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2025-08-01","abstract_html":"Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2025-08-01","abstract_has_math":false,"creators":["Xie, Tengyang"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Jiang, Nan","Forsyth, David","Srikant, Rayadurgam","Brunskill, Emma","Chen, Xi"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2023,"date_issued":"2023-08","date_published":"2023-08","updated_at":"2026-07-22T22:24:57Z","subjects":["Artificial Intelligence","Machine Learning","Reinforcement Learning","Statistical Learning Theory"],"languages":["en","eng"],"rights":["Copyright 2023 Tengyang Xie"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/121258","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Jiang, Nan","Forsyth, David","Srikant, Rayadurgam","Brunskill, Emma","Chen, Xi"]},{"key":"dc:creator","label":"Author","values":["Xie, Tengyang"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2023-08","2023-07-13"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Artificial Intelligence","Machine Learning","Reinforcement Learning","Statistical Learning Theory"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2023 Tengyang Xie"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/121258"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2025-08-01","The student, Tengyang Xie, accepted the attached license on 2023-07-12 at 00:04.","The student, Tengyang Xie, submitted this Dissertation for approval on 2023-07-12 at 00:13.","This Dissertation was approved for publication on 2023-07-13 at 16:37.","DSpace SAF Submission Ingestion Package generated from Vireo submission #19645 on 2023-12-04 at 17:34:26","Reinforcement Learning (RL) has displayed its formidable capabilities across a plethora of complex challenges, including mastering the game of Go (Silver et al., 2016, 2017) and strategically navigating video games such as StarCraft II (Vinyals et al., 2019). Despite these triumphs in simulated environments, the real-world impact of reinforcement learning is yet to fully materialize, in stark contrast to its simulated successes. This dissertation sets out to take steps towards a lofty, long-term objective: to transfer the impressive achievements of reinforcement learning from simulated environments into real-world scenarios. Unlike supervised learning, which is a paradigm naturally attuned to data-driven applications and already prevalent in a multitude of real-world settings, reinforcement learning is innately interactive. This trait poses one of the most significant impediments to the broader real-world adoption of reinforcement learning. The interactive nature of reinforcement learning necessitates an iterative learning procedure involving the continuous collection of experiential data from the environment, as well as learning from the collected data, primarily by implementing the most recently learned policy for both evaluation and improvement. This degree of online interaction, often necessitating a high degree of adaptivity or policy switching, is frequently unfeasible in real-world contexts due to the expensive or even risky data collection procedures involved, such as in robotics, autonomous driving, or healthcare. Addressing this issue of low adaptivity has been a longstanding focus in reinforcement learning research, with numerous studies examining off-policy learning, which permits a divergence between the data-collection policy and the target policy. Yet, most off-policy reinforcement learning techniques typically struggle with the problem of so-called offline reinforcement learning, which studies reinforcement learning using purely offline data. This dissertation, therefore, targets on a complete solution for reinforcement learning with offline data. The proposed approaches are primarily motivated by the aforementioned challenges and aim to enable the data-driven learning paradigm of reinforcement learning from both a theoretical and an algorithmic perspective. To attain such goal, this dissertation investigates the following three main topics: Part I: Foundations of Offline Reinforcement Learning. This part answers several fundamental questions about offline RL, including the minimal approximation assumption for conducting offline RL, as well as the fundamental algorithmic concept of offline RL. Part II: Algorithms of Offline Reinforcement Learning. This part presents algorithms for offline RL that are both provably efficient and capable of achieving strong performance. In addition, these algorithms also attain desirable properties for practical applications, especially for achieving the best of both worlds from offline RL and imitation learning. Part III: Bridging Offline and Online Reinforcement Learning. The last part of this dissertation goes beyond the purely offline setting and works towards bridging sample-efficient online and offline reinforcement learning. This part investigates the perspectives of bridging algorithmic concepts and learnability conditions for online and offline RL."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Reinforcement learning with offline data: foundations and algorithms"]}]}],"canonical_facts":{"dc:contributor":["Jiang, Nan","Forsyth, David","Srikant, Rayadurgam","Brunskill, Emma","Chen, Xi"],"dc:creator":["Xie, Tengyang"],"dc:date":["2023-08","2023-07-13"],"dc:description":["Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2025-08-01","The student, Tengyang Xie, accepted the attached license on 2023-07-12 at 00:04.","The student, Tengyang Xie, submitted this Dissertation for approval on 2023-07-12 at 00:13.","This Dissertation was approved for publication on 2023-07-13 at 16:37.","DSpace SAF Submission Ingestion Package generated from Vireo submission #19645 on 2023-12-04 at 17:34:26","Reinforcement Learning (RL) has displayed its formidable capabilities across a plethora of complex challenges, including mastering the game of Go (Silver et al., 2016, 2017) and strategically navigating video games such as StarCraft II (Vinyals et al., 2019). Despite these triumphs in simulated environments, the real-world impact of reinforcement learning is yet to fully materialize, in stark contrast to its simulated successes. This dissertation sets out to take steps towards a lofty, long-term objective: to transfer the impressive achievements of reinforcement learning from simulated environments into real-world scenarios. Unlike supervised learning, which is a paradigm naturally attuned to data-driven applications and already prevalent in a multitude of real-world settings, reinforcement learning is innately interactive. This trait poses one of the most significant impediments to the broader real-world adoption of reinforcement learning. The interactive nature of reinforcement learning necessitates an iterative learning procedure involving the continuous collection of experiential data from the environment, as well as learning from the collected data, primarily by implementing the most recently learned policy for both evaluation and improvement. This degree of online interaction, often necessitating a high degree of adaptivity or policy switching, is frequently unfeasible in real-world contexts due to the expensive or even risky data collection procedures involved, such as in robotics, autonomous driving, or healthcare. Addressing this issue of low adaptivity has been a longstanding focus in reinforcement learning research, with numerous studies examining off-policy learning, which permits a divergence between the data-collection policy and the target policy. Yet, most off-policy reinforcement learning techniques typically struggle with the problem of so-called offline reinforcement learning, which studies reinforcement learning using purely offline data. This dissertation, therefore, targets on a complete solution for reinforcement learning with offline data. The proposed approaches are primarily motivated by the aforementioned challenges and aim to enable the data-driven learning paradigm of reinforcement learning from both a theoretical and an algorithmic perspective. To attain such goal, this dissertation investigates the following three main topics: Part I: Foundations of Offline Reinforcement Learning. This part answers several fundamental questions about offline RL, including the minimal approximation assumption for conducting offline RL, as well as the fundamental algorithmic concept of offline RL. Part II: Algorithms of Offline Reinforcement Learning. This part presents algorithms for offline RL that are both provably efficient and capable of achieving strong performance. In addition, these algorithms also attain desirable properties for practical applications, especially for achieving the best of both worlds from offline RL and imitation learning. Part III: Bridging Offline and Online Reinforcement Learning. The last part of this dissertation goes beyond the purely offline setting and works towards bridging sample-efficient online and offline reinforcement learning. This part investigates the perspectives of bridging algorithmic concepts and learnability conditions for online and offline RL."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/121258"],"dc:language":["en","eng"],"dc:rights":["Copyright 2023 Tengyang Xie"],"dc:subject":["Artificial Intelligence","Machine Learning","Reinforcement Learning","Statistical Learning Theory"],"dc:title":["Reinforcement learning with offline data: foundations and algorithms"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:57Z"}