{"id":{"repo_id":"vt","oai_identifier":"oai:vtechworks.lib.vt.edu:10919/139710"},"canonical_url":"https://search.dev.ndltd.org/etd/vt/oai:vtechworks.lib.vt.edu:10919/139710","repository":{"repo_id":"vt","name":"Virginia Tech","base_url":"https://vtechworks.lib.vt.edu/oai/request"},"display":{"title":"Toward Predictable and Efficient Deep Neural Network Inference on Graphics Processing Units","abstract":"GPUs dominate DNN inference but remain difficult to control predictably under multi-tenant load. This thesis presents a practical, end-to-end approach for predictable, efficient single-GPU inference built around a closed loop of predict → allocate → power-tune. First, we introduce SGPRS, a spatio–temporal scheduler that combines spatial partitioning with stream-based temporal concurrency and explicit staging for coarse preemption and partition reuse without costly reconfiguration. Second, we develop GRAIL, a lightweight online predictor of latency and throughput under varying TPC (SM-group) and clock settings, and GRAIL-A, a zero-overhead allocator that turns those predictions into fast TPC and clock decisions. Third, we design SAGE, a power-aware runtime that coordinates DVFS and TPC control via a central partitioner and per-tenant local schedulers. Across diverse CNN and Transformer models and workload mixes, the system improves total throughput, tightens P95 and P99 latency, and reduces energy compared to temporal-only, spatial-only, and framework baselines, while preserving deadline behavior. The result is a resource-aware, empirically validated path to predictable multi-tenant inference on a single NVIDIA GPU.","abstract_html":"GPUs dominate DNN inference but remain difficult to control predictably under multi-tenant load. This thesis presents a practical, end-to-end approach for predictable, efficient single-GPU inference built around a closed loop of predict → allocate → power-tune. First, we introduce SGPRS, a spatio–temporal scheduler that combines spatial partitioning with stream-based temporal concurrency and explicit staging for coarse preemption and partition reuse without costly reconfiguration. Second, we develop GRAIL, a lightweight online predictor of latency and throughput under varying TPC (SM-group) and clock settings, and GRAIL-A, a zero-overhead allocator that turns those predictions into fast TPC and clock decisions. Third, we design SAGE, a power-aware runtime that coordinates DVFS and TPC control via a central partitioner and per-tenant local schedulers. Across diverse CNN and Transformer models and workload mixes, the system improves total throughput, tightens P95 and P99 latency, and reduces energy compared to temporal-only, spatial-only, and framework baselines, while preserving deadline behavior. The result is a resource-aware, empirically validated path to predictable multi-tenant inference on a single NVIDIA GPU.","abstract_has_math":false,"creators":["Fakhim Babaei, Amir"],"institution":"Virginia Tech","degree_name":"Doctor of Philosophy","degree_level":"doctoral","degree_discipline":"Computer Engineering","degree_department":"Electrical and Computer Engineering","school":null,"contributors":[],"advisors":[],"committee_chairs":["Chantem, Thidapat"],"committee_members":["Wang, Yue J.","Stavrou, Angelos","Tilevich, Eli","Dimarino, Christina Marie"],"year":2025,"date_issued":"2025-11-20","date_published":"2025-11-20","updated_at":"2026-07-22T22:20:27Z","subjects":["GPU","Real-Time Systems","WCET","Inference"],"languages":["en"],"rights":["In Copyright"],"rights_urls":["http://rightsstatements.org/vocab/InC/1.0/"],"identifier_entries":[{"key":"dc:identifier.other","label":"Dc Identifier Other","values":["vt_gsexam:45061"],"render_values":[{"text":"vt_gsexam:45061","href":null,"code":true}]}]},"links":{"outbound_url":"https://hdl.handle.net/10919/139710","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.committeechair","label":"Committee Chair","values":["Chantem, Thidapat"]},{"key":"dc:contributor.committeemember","label":"Committee Member","values":["Wang, Yue J.","Stavrou, Angelos","Tilevich, Eli","Dimarino, Christina Marie"]},{"key":"dc:contributor.department","label":"Department","values":["Electrical and Computer Engineering"]},{"key":"dc:creator","label":"Author","values":["Fakhim Babaei, Amir"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2025-11-21T09:00:11Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2025-11-21T09:00:11Z"]},{"key":"dc:date.issued","label":"Date","values":["2025-11-20"]},{"key":"dc:publisher","label":"Institution","values":["Virginia Tech"]},{"key":"dc:type","label":"Dc Type","values":["Dissertation"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Engineering"]},{"key":"thesis:degree_level","label":"Degree Level","values":["doctoral"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Doctor of Philosophy"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["Virginia Polytechnic Institute and State University"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["GPU","Real-Time Systems","WCET","Inference"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["In Copyright"]},{"key":"dc:rights.uri","label":"Rights URI","values":["http://rightsstatements.org/vocab/InC/1.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.other","label":"Dc Identifier Other","values":["vt_gsexam:45061"]},{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/10919/139710"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["GPUs dominate DNN inference but remain difficult to control predictably under multi-tenant load. This thesis presents a practical, end-to-end approach for predictable, efficient single-GPU inference built around a closed loop of predict → allocate → power-tune. First, we introduce SGPRS, a spatio–temporal scheduler that combines spatial partitioning with stream-based temporal concurrency and explicit staging for coarse preemption and partition reuse without costly reconfiguration. Second, we develop GRAIL, a lightweight online predictor of latency and throughput under varying TPC (SM-group) and clock settings, and GRAIL-A, a zero-overhead allocator that turns those predictions into fast TPC and clock decisions. Third, we design SAGE, a power-aware runtime that coordinates DVFS and TPC control via a central partitioner and per-tenant local schedulers. Across diverse CNN and Transformer models and workload mixes, the system improves total throughput, tightens P95 and P99 latency, and reduces energy compared to temporal-only, spatial-only, and framework baselines, while preserving deadline behavior. The result is a resource-aware, empirically validated path to predictable multi-tenant inference on a single NVIDIA GPU."]},{"key":"dc:description.abstractgeneral","label":"General Abstract","values":["GPUs make AI fast, but when many AI tasks share one GPU the results can be unpredictable and wasteful. This work builds a simple loop—predict how long tasks will take, allocate the right slice of the GPU and clock speed, and power-tune to avoid waste—that lets multiple models run together without surprises. We split networks into stages so they can pause and resume cleanly, reuse GPU partitions instead of reconfiguring them, and adjust power and speed on the fly. In tests with many kinds of models and loads, the system ran more jobs, met timing more reliably, and used less energy than common setups."]},{"key":"dc:description.degree","label":"Dc Description Degree","values":["Doctor of Philosophy"]},{"key":"dc:format.medium","label":"Dc Format Medium","values":["ETD"]},{"key":"dc:title","label":"Title","values":["Toward Predictable and Efficient Deep Neural Network Inference on Graphics Processing Units"]}]}],"canonical_facts":{"dc:contributor.committeechair":["Chantem, Thidapat"],"dc:contributor.committeemember":["Wang, Yue J.","Stavrou, Angelos","Tilevich, Eli","Dimarino, Christina Marie"],"dc:contributor.department":["Electrical and Computer Engineering"],"dc:creator":["Fakhim Babaei, Amir"],"dc:date.accessioned":["2025-11-21T09:00:11Z"],"dc:date.available":["2025-11-21T09:00:11Z"],"dc:date.issued":["2025-11-20"],"dc:description.abstract":["GPUs dominate DNN inference but remain difficult to control predictably under multi-tenant load. This thesis presents a practical, end-to-end approach for predictable, efficient single-GPU inference built around a closed loop of predict → allocate → power-tune. First, we introduce SGPRS, a spatio–temporal scheduler that combines spatial partitioning with stream-based temporal concurrency and explicit staging for coarse preemption and partition reuse without costly reconfiguration. Second, we develop GRAIL, a lightweight online predictor of latency and throughput under varying TPC (SM-group) and clock settings, and GRAIL-A, a zero-overhead allocator that turns those predictions into fast TPC and clock decisions. Third, we design SAGE, a power-aware runtime that coordinates DVFS and TPC control via a central partitioner and per-tenant local schedulers. Across diverse CNN and Transformer models and workload mixes, the system improves total throughput, tightens P95 and P99 latency, and reduces energy compared to temporal-only, spatial-only, and framework baselines, while preserving deadline behavior. The result is a resource-aware, empirically validated path to predictable multi-tenant inference on a single NVIDIA GPU."],"dc:description.abstractgeneral":["GPUs make AI fast, but when many AI tasks share one GPU the results can be unpredictable and wasteful. This work builds a simple loop—predict how long tasks will take, allocate the right slice of the GPU and clock speed, and power-tune to avoid waste—that lets multiple models run together without surprises. We split networks into stages so they can pause and resume cleanly, reuse GPU partitions instead of reconfiguring them, and adjust power and speed on the fly. In tests with many kinds of models and loads, the system ran more jobs, met timing more reliably, and used less energy than common setups."],"dc:description.degree":["Doctor of Philosophy"],"dc:format.medium":["ETD"],"dc:identifier.other":["vt_gsexam:45061"],"dc:identifier.uri":["https://hdl.handle.net/10919/139710"],"dc:language.iso":["en"],"dc:publisher":["Virginia Tech"],"dc:rights":["In Copyright"],"dc:rights.uri":["http://rightsstatements.org/vocab/InC/1.0/"],"dc:subject":["GPU","Real-Time Systems","WCET","Inference"],"dc:title":["Toward Predictable and Efficient Deep Neural Network Inference on Graphics Processing Units"],"dc:type":["Dissertation"],"thesis:degree_discipline":["Computer Engineering"],"thesis:degree_level":["doctoral"],"thesis:degree_name":["Doctor of Philosophy"],"thesis:institution_name":["Virginia Polytechnic Institute and State University"]},"updated_at":"2026-07-22T22:20:27Z"}