Enhancing hybrid cloud ray clusters: automated data management and innovative networking for efficient HPC cloud bursting

The increasing reliance on hybrid systems that combine High-Performance Computing (HPC) and Cloud computing reflects their ability to manage workload surges and reduce users' queuing time. This thesis presents the development of an innovative HPC-Cloud bursting system, which leverages the Ray open-source distributed framework. Our system features an advanced automated data management mechanism and a unique dynamic label-based scheduling algorithm that intelligently manages data dependencies and minimizes data transmission overheads. My personal contributions to this project were critical in several areas: I designed and implemented network solutions and infrastructure as code, significantly simplifying the deployment and operational management of the hybrid system while also reducing costs. Additionally, I played an important role in optimizing the system's scheduler to improve stability and contributed directly to a performance boost. Our benchmarks, which focus on machine learning model training and image processing tasks, demonstrate our system's enhanced efficiency, achieving up to 20\% improvement when compared to traditional methods.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Zhu, Zhongbo

Contributors dc:contributor

• Kindratenko, Volodymyr

• Distributed Systems
• Cloud Computing
• High Performance Computing
• Distributed Machine Learning

Statement dc:rights

• Copyright 2024 Zhongbo Zhu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124427

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124427