Optimizing data movement in cloud-bursting HPC environments through dynamic labeling and prefetching strategies

Hybrid High-Performance Computing-Cloud systems are gaining popularity among researchers for their ability to handle sudden demand spikes, resulting in accelerated turnaround times for High-Performance Computing (HPC) tasks. However, deploying workloads on such systems presents challenges, particularly in data migration across HPC clusters and the Cloud, and the lack of support in existing schedulers for hybrid environments. To address these issues, we present an HPC-Cloud bursting system leveraging Ray, an open-source distributed framework. Our system seamlessly integrates automated data management with data prefetching and learning-based scheduling at the function level. In this project, my primary focus was on implementing dynamic labeling within the Ray framework, enabling adjustments and modifications to node labels during the runtime. This dynamic labeling is then seamlessly integrated with the workload scheduler to facilitate strategic data prefetching to the most suitable nodes. Additionally, I played a pivotal role in enhancing the compatibility of our system with Cloud Storage Service, thereby expanding its versatility and usability. We assess the effectiveness of our framework by employing two prevalent workloads: machine learning model training and image processing. Our findings reveal that our system consistently yields advantages across diverse data locations and network speeds when compared to the manual data fetching baseline for both workloads.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Tao, Huili

Contributors dc:contributor

• Kindratenko, Volodymyr

• Hpc
• Cloud Bursting
• Data Movement

Statement dc:rights

• Copyright 2024 Huili Tao

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124598

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124598