Learning-based scheduling for ray-based Hybrid HPC-Cloud Systems

Hybrid HPC-Cloud systems are becoming increasingly popular within the scientific community for their ability to efficiently manage sudden increases in demand, thus improving the processing times of HPC workloads. However, current systems lack efficient workload scheduling strategies to suit these hybrid environments and face considerable deployment challenges due to intricate configurations required, particularly concerning data transfer between HPC and the cloud. To address these issues, we have developed an innovative HPC-Cloud bursting system using Ray, a well-known open-source distributed framework. Our system adopts a learning-based scheduling approach at the function level through a dynamic label-based architecture and automatically manages data movement between the cloud and HPC. Specifically, our scheduler proactively prefetches data based on anticipated demand and analyzes patterns of data movement and task execution to inform future scheduling decisions. Our system significantly improves the processing times of HPC workloads by hiding data transfer time and employing high-quality, learning-based scheduling decisions. We evaluated our system with two different workloads: machine learning model training and image processing. We conducted performance comparisons using conventional data retrieval methods and the default Ray scheduler under various network conditions and storage configurations. Our findings consistently show that our system significantly outperforms traditional methods in every tested scenario.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Lu, Yicheng

Contributors dc:contributor

• Kindratenko, Volodymyr

• Cloud Bursting
• Hpc
• Data Movement
• Scheduling

Statement dc:rights

• Copyright 2024 Yicheng Lu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124562

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124562