Cloud-bursting and autoscaling for Python-native scientific workflows

In this work, the Ray framework is extended to enable automatic scaling of workloads on high-performance computing (HPC) clusters managed by SLURM and bursting to Cloud managed by Kubernetes. Compared to existing HPC-Cloud convergence solutions, this framework demonstrates advantages in several aspects: users can provide their own Cloud resource, framework provides the Python-level abstraction that does not require users to interact with job submission systems, and it allows a single Python-based parallel workload to be run concurrently across an HPC cluster and a Cloud. Applications in Electronic Design Automation and distributed Machine Learning are used to demonstrate the functionality of this solution in scaling the workload on an on-premises HPC system and automatically bursting to a public Cloud when running out of allocated HPC resources. The thesis focuses on describing the initial implementation and demonstrating novel functionality of the proposed framework, as well as identifying practical considerations and limitations for using Cloud bursting mode. The code of this framework has been open-sourced.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Liu, Tingkai

Contributors dc:contributor

• Kindratenko, Volodymyr

• Hpc
• Cloud
• Cloud-bursting

Statement dc:rights

• Copyright 2023 Tingkai Liu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/120266

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F120266