{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/124562"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/124562","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Learning-based scheduling for ray-based Hybrid HPC-Cloud Systems","abstract":"Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2026-05-01","abstract_html":"Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2026-05-01","abstract_has_math":false,"creators":["Lu, Yicheng"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Kindratenko, Volodymyr"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024-05","date_published":"2024-05","updated_at":"2026-07-22T22:25:02Z","subjects":["Cloud Bursting","Hpc","Data Movement","Scheduling"],"languages":["en","eng"],"rights":["Copyright 2024 Yicheng Lu"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/124562","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Kindratenko, Volodymyr"]},{"key":"dc:creator","label":"Author","values":["Lu, Yicheng"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2024-05","2024-04-30"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Cloud Bursting","Hpc","Data Movement","Scheduling"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2024 Yicheng Lu"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/124562"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2026-05-01","The student, Yicheng Lu, accepted the attached license on 2024-04-30 at 09:32.","The student, Yicheng Lu, submitted this Thesis for approval on 2024-04-30 at 09:38.","This Thesis was approved for publication on 2024-04-30 at 11:32.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20560 on 2024-09-16 at 00:44:24","Hybrid HPC-Cloud systems are becoming increasingly popular within the scientific community for their ability to efficiently manage sudden increases in demand, thus improving the processing times of HPC workloads. However, current systems lack efficient workload scheduling strategies to suit these hybrid environments and face considerable deployment challenges due to intricate configurations required, particularly concerning data transfer between HPC and the cloud. To address these issues, we have developed an innovative HPC-Cloud bursting system using Ray, a well-known open-source distributed framework. Our system adopts a learning-based scheduling approach at the function level through a dynamic label-based architecture and automatically manages data movement between the cloud and HPC. Specifically, our scheduler proactively prefetches data based on anticipated demand and analyzes patterns of data movement and task execution to inform future scheduling decisions. Our system significantly improves the processing times of HPC workloads by hiding data transfer time and employing high-quality, learning-based scheduling decisions. We evaluated our system with two different workloads: machine learning model training and image processing. We conducted performance comparisons using conventional data retrieval methods and the default Ray scheduler under various network conditions and storage configurations. Our findings consistently show that our system significantly outperforms traditional methods in every tested scenario."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Learning-based scheduling for ray-based Hybrid HPC-Cloud Systems"]}]}],"canonical_facts":{"dc:contributor":["Kindratenko, Volodymyr"],"dc:creator":["Lu, Yicheng"],"dc:date":["2024-05","2024-04-30"],"dc:description":["Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2026-05-01","The student, Yicheng Lu, accepted the attached license on 2024-04-30 at 09:32.","The student, Yicheng Lu, submitted this Thesis for approval on 2024-04-30 at 09:38.","This Thesis was approved for publication on 2024-04-30 at 11:32.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20560 on 2024-09-16 at 00:44:24","Hybrid HPC-Cloud systems are becoming increasingly popular within the scientific community for their ability to efficiently manage sudden increases in demand, thus improving the processing times of HPC workloads. However, current systems lack efficient workload scheduling strategies to suit these hybrid environments and face considerable deployment challenges due to intricate configurations required, particularly concerning data transfer between HPC and the cloud. To address these issues, we have developed an innovative HPC-Cloud bursting system using Ray, a well-known open-source distributed framework. Our system adopts a learning-based scheduling approach at the function level through a dynamic label-based architecture and automatically manages data movement between the cloud and HPC. Specifically, our scheduler proactively prefetches data based on anticipated demand and analyzes patterns of data movement and task execution to inform future scheduling decisions. Our system significantly improves the processing times of HPC workloads by hiding data transfer time and employing high-quality, learning-based scheduling decisions. We evaluated our system with two different workloads: machine learning model training and image processing. We conducted performance comparisons using conventional data retrieval methods and the default Ray scheduler under various network conditions and storage configurations. Our findings consistently show that our system significantly outperforms traditional methods in every tested scenario."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/124562"],"dc:language":["en","eng"],"dc:rights":["Copyright 2024 Yicheng Lu"],"dc:subject":["Cloud Bursting","Hpc","Data Movement","Scheduling"],"dc:title":["Learning-based scheduling for ray-based Hybrid HPC-Cloud Systems"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:02Z"}