Configurable Hierarchical Allreduce Algorithms for HPC Systems

Collective communication is a critical component of large scale scientific applications across all message size ranges. However, optimizing performance in the small to medium message regime is especially challenging because both bandwidth and latency influence efficiency. Many workloads, including Krylov based iterative solvers, multidimensional fft based simulations, and modern machine learning applications, issue thousands of Allreduce, Allgather, and Reduce Scatter operations during each run. Achieving efficient performance in this regime requires algorithms that explicitly account for the two tier communication hierarchy present in current high performance systems, where intra node communication differs significantly from inter node communication. This thesis introduces a configurable hierarchical design for the Allgather and Reduce Scatter collectives, which serve as the building blocks for a parameterized Allreduce. The approach is controlled by two variables: the radix k of the recursive exchange pattern and the batch size b that groups processes to exploit intra node shared memory communication. Allreduce is implemented following the Rabenseifner method, which decomposes the operation into a Reduce Scatter followed by an Allgather. By parameterizing these two underlying collectives and aligning their communication structure, the design reduces unnecessary data movement and yields a flexible hierarchical Allreduce. This approach naturally produces faster implementations of Allgather and Reduce Scatter as direct byproducts of the design. The central idea is to expose k and b as tunable parameters so that the algorithms can adapt to a variety of network topologies and hierarchies, rather than relying on a single communication pattern. We evaluate the proposed collectives on three large scale supercomputers with significantly different architectures and interconnects: Polaris and Aurora at Argonne National Laboratory, and Fugaku at RIKEN. These platforms span fat tree like, high radix Slingshot, and torus mesh networks, providing a broad view of architectural diversity. Across all systems, the hierarchical collectives achieve substantial speedups over vendor optimized baselines, particularly for small and medium message sizes. The results demonstrate that parameterized hierarchical communication is an effective strategy for improving robustness, portability, and scalability in core mpi collectives used throughout scientific computing and machine learning.

Author dc:creator

• Valentino Guerrini (24399479)

• Computer Science

Statement dc:rights

• In Copyright

DOI dc:identifier

https://doi.org/10.25417/uic.32993918.v1

OAI identifier oai:identifier

oai:figshare.com:article/32993918