{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/114113"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/114113","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Reinforcement learning using reservoir computing for soft robotic control: A bio-inspired system","abstract":"Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2023-12-01","abstract_html":"Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2023-12-01","abstract_has_math":false,"creators":["Shivam, Keshav"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Gazzola, Mattia","Chowdhary, Girish"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2022,"date_issued":"2022-04-29T21:58:42Z","date_published":"2022-04-29T21:58:42Z","updated_at":"2026-07-22T22:24:54Z","subjects":["Computer science"],"languages":["en","eng"],"rights":["Copyright 2021 Keshav Shivam"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/114113","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Gazzola, Mattia","Chowdhary, Girish"]},{"key":"dc:creator","label":"Author","values":["Shivam, Keshav"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2022-04-29T21:58:42Z","2024-04-29T21:58:46Z","2021-12","2021-12-09"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computer science"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2021 Keshav Shivam"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/114113"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2023-12-01","The student, Keshav Shivam, accepted the attached license on 2021-12-09 at 11:12.","The student, Keshav Shivam, submitted this Thesis for approval on 2021-12-09 at 11:34.","This Thesis was approved for publication on 2021-12-09 at 13:34.","DSpace SAF Submission Ingestion Package generated from Vireo submission #17433 on 2022-04-29 at 16:10:47","Made available in DSpace on 2022-04-29T21:58:42Z (GMT). No. of bitstreams: 2 SHIVAM-THESIS-2021.pdf: 22018320 bytes, checksum: b5afe75a499f6b7aecaf4aa6e937a664 (MD5) LICENSE.txt: 4210 bytes, checksum: aa68c84ff9ce8ec105645985ccb29d3b (MD5) Previous issue date: 2021-12-09","Embargo set by: Seth Robbins for item 123478 Lift date: 2024-04-29T21:58:46Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Author requested closed access (OA after 2yrs) in Vireo ETD system","Limited","Modern approaches in machine learning and artificial intelligence are dominated by deep learning. Although inspired by the brain, these network architectures are not biologically plausible. In contrast, the reservoir computing paradigm has a single sparsely and recurrently connected hidden layer, with the linear readout layer being the only learned parameter. We apply reservoir computing in a reinforcement learning context to actuate a soft, slender muscular arm. The arm must track a moving target in a partially observable environment. Soft robots present an especially challenging test bed for reinforcement learning due to their nonlinear, continuum dynamics. We propose learning strategies for two classes of reservoirs: Echo State Networks (ESNs) built using tanh neurons and Liquid State Machines (LSMs) built using Leaky Integrate and Fire (LIF) neurons. Unlike traditional activation functions, LIF neurons provide discrete spike trains with respect to time, and LSM-like structures are found in vivo. Crucially, we prohibit any additional feedforward layers, making the reservoir the sole neural computing unit. Our ESN policy significantly outperforms the state-of-the-art algorithm Proximal Policy Optimization (PPO) and our LSM policy matches PPO. Finally, we deploy a LSM directly on neuromorphic hardware, opening up opportunities for energy efficient reinforcement learning and robotic control. End-to-end, our soft robot controlled with a spiking reservoir is a novel bio-inspired system."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Reinforcement learning using reservoir computing for soft robotic control: A bio-inspired system"]}]}],"canonical_facts":{"dc:contributor":["Gazzola, Mattia","Chowdhary, Girish"],"dc:creator":["Shivam, Keshav"],"dc:date":["2022-04-29T21:58:42Z","2024-04-29T21:58:46Z","2021-12","2021-12-09"],"dc:description":["Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2023-12-01","The student, Keshav Shivam, accepted the attached license on 2021-12-09 at 11:12.","The student, Keshav Shivam, submitted this Thesis for approval on 2021-12-09 at 11:34.","This Thesis was approved for publication on 2021-12-09 at 13:34.","DSpace SAF Submission Ingestion Package generated from Vireo submission #17433 on 2022-04-29 at 16:10:47","Made available in DSpace on 2022-04-29T21:58:42Z (GMT). No. of bitstreams: 2 SHIVAM-THESIS-2021.pdf: 22018320 bytes, checksum: b5afe75a499f6b7aecaf4aa6e937a664 (MD5) LICENSE.txt: 4210 bytes, checksum: aa68c84ff9ce8ec105645985ccb29d3b (MD5) Previous issue date: 2021-12-09","Embargo set by: Seth Robbins for item 123478 Lift date: 2024-04-29T21:58:46Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Author requested closed access (OA after 2yrs) in Vireo ETD system","Limited","Modern approaches in machine learning and artificial intelligence are dominated by deep learning. Although inspired by the brain, these network architectures are not biologically plausible. In contrast, the reservoir computing paradigm has a single sparsely and recurrently connected hidden layer, with the linear readout layer being the only learned parameter. We apply reservoir computing in a reinforcement learning context to actuate a soft, slender muscular arm. The arm must track a moving target in a partially observable environment. Soft robots present an especially challenging test bed for reinforcement learning due to their nonlinear, continuum dynamics. We propose learning strategies for two classes of reservoirs: Echo State Networks (ESNs) built using tanh neurons and Liquid State Machines (LSMs) built using Leaky Integrate and Fire (LIF) neurons. Unlike traditional activation functions, LIF neurons provide discrete spike trains with respect to time, and LSM-like structures are found in vivo. Crucially, we prohibit any additional feedforward layers, making the reservoir the sole neural computing unit. Our ESN policy significantly outperforms the state-of-the-art algorithm Proximal Policy Optimization (PPO) and our LSM policy matches PPO. Finally, we deploy a LSM directly on neuromorphic hardware, opening up opportunities for energy efficient reinforcement learning and robotic control. End-to-end, our soft robot controlled with a spiking reservoir is a novel bio-inspired system."],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/114113"],"dc:language":["en","eng"],"dc:rights":["Copyright 2021 Keshav Shivam"],"dc:subject":["Computer science"],"dc:title":["Reinforcement learning using reservoir computing for soft robotic control: A bio-inspired system"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:54Z"}