Designing scalable large-scale storage-based GNN framework by efficiently leveraging heterogeneous hardware resources

Graph Neural Networks (GNNs) are widely used today in applications such as recommendation systems, fraud detection, and node/link classification tasks. As real-world graphs and embeddings used for GNN training continue to grow in scale, their memory footprint often exceeds the memory capacities of GPUs, creating a bottleneck in efficient training. Traditional GNN training frameworks address limited memory by either storing feature data in external storage and fetching it on-demand or sharding the graph across multiple GPUs and transferring data as needed. However, the first approach suffers from high storage latency, while the second is burdened by the high computational costs of graph partitioning, excessive inter-GPU communication, and increased total cost of ownership. To address these challenges, this dissertation introduces three storage-based GNN frameworks—GIDS, LSM-GNN, and SSD-GNN—that span single-GPU, multi-GPU, and multi-node environments, respectively. GIDS accelerates single-GPU GNN training by leveraging GPU thread parallelism to hide storage latency. LSM-GNN extends this to a multi-GPU setting by implementing a system-wide shared cache using NVLink, optimizing memory bandwidth and cache hit rates without graph partitioning. SSD-GNN further scales GNN training across multiple nodes by integrating GPU-initiated direct storage access and a distributed caching protocol to reduce data movement overhead across nodes. These frameworks effectively utilize heterogeneous hardware resources such as SSDs, CPU memory, GPU parallelism, and GPU PCIe bandwidth to optimize data transfer operations and hide storage latency. Prototypes of these systems demonstrate significant improvements in GNN training performance and reduced total cost of ownership compared to state-of-the-art systems, marking a new pathway for scalable, storage-efficient GNN training across a range of compute environments.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Park, Jeongmin

Contributors dc:contributor

• Hwu, Wen-mei
• Patel, Sanjay
• Chen, Deming
• Lumetta, Steven S

• Storage
• Large-scale Ml
• Distributed System
• Gpu
• Parallel Computing
• Gnn

Statement dc:rights

• Copyright 2024 Jeongmin Park

Language dc:language

eng, en

Handle dc:identifier

https://hdl.handle.net/2142/127395

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F127395