Analysis of Memory Access Patterns for Large Language Model Inference

The use of tiered heterogeneous memory systems in HPC workloads is growing in popularity as the increasing memory requirements for these workloads outpace the decline in the cost- per-gigabyte of fast DRAM; however, the Linux kernel has no intelligent strategy to manage these tiered memory systems. Because of this limitation, a great deal of research has been conducted to identify policies that make efficient use of these systems. Much of this prior research focuses on deep learning tasks, while only a few focus on inference for large models. The training and inference workloads for the same type of model are quite different: in training, the task is to continuously update the weights matrices with knowledge gained from each training datum, while in inference, the workload only reads from the weights. Training for neural networks also involves accesses in reverse order to what is used in inference, in a training technique called backpropagation. This thesis presents a memory access pattern heatmap tool that can track evolving access patterns through the lifetime of a workload. This tool is applied to llama.cpp, an LLM inference tool, to identify memory access patterns between remote and local NUMA nodes. The thesis then explores two basic NUMA page placement strategies, where all memory is bound to either the local or remote NUMA nodes to identify the impact of poor NUMA policies on performance and compares them to the default Linux strategy.

Name thesis:degree_name

Master of Science

Level thesis:degree_level

masters

Discipline thesis:degree_discipline

Computer Science & Applications

Department dc:contributor.department

Computer Science and#38; Applications

Grantor dc:publisher

Virginia Tech

Year dc:date.issued

2025

Author dc:creator

• Fisher, Max Henry

Chair dc:contributor.committeechair

• Nikolopoulos, Dimitrios S.

Committee members dc:contributor.committeemember

• Back, Godmar Volker
• Li, Huaicheng

• NUMA
• Page Placement
• Memory Access Patterns
• High Performance Computing
• LLM Inference

Statement dc:rights

• Creative Commons Attribution-ShareAlike 4.0 International

Licence dc:rights.uri

• http://creativecommons.org/licenses/by-sa/4.0/

Language dc:language.iso

en

Dc Identifier Other

vt_gsexam:43726

OAI identifier oai:identifier

oai:vtechworks.lib.vt.edu:10919/135946