Chasing the “tail at scale”: toward cloud-native architectures

Cloud computing is undergoing a radical transformation with the emergence of lightweight cloud-native computing paradigms, such as microservices and serverless computing. Users build their applications by combining services, benefiting from a simplified programming model and fine-grained billing. At the same time, providers consolidate many services into a smaller number of servers, improving the utilization of their infrastructure. However, the detailed characterization of cloud-native environments presented in this thesis shows that these workloads differ significantly from traditional monolithic applications. They execute services that run for short times, exhibit bursty invocation patterns, and have frequent I/O operations that cause context switches. In addition to their core logic, services also execute many auxiliary operations known as datacenter tax, such as data serialization and encryption. Finally, services have stringent tail latency bounds, requiring the slowest requests to complete within a strict deadline. These characteristics result in significant inefficiencies in performance, energy, and resource utilization when cloud-native workloads run on conventional servers with conventional software stacks, negating the paradigm’s potential benefits. The goal of this thesis is to design hardware platforms and software stacks that enable the execution of cloud-native workloads with orders of magnitude better efficiency. The first part of the thesis designs a new hardware stack for cloud-native services. It introduces µManycore, a CPU architecture that minimizes the tail latency of cloud-native services. The thesis then extends the architecture with HardHarvest to boost utilization via hardware-based core harvesting, and refines the microarchitecture with Mosaic for better performance under frequent context switches. Finally, this thesis integrates on-package accelerators into the architecture and proposes AccelFlow, a framework that enables fine-grained, low-overhead orchestration of accelerators to reduce the datacenter tax in cloud-native environments. To maximize the efficiency of the proposed hardware architecture, the second part of the thesis builds a full software stack that is tightly co-designed with the hardware. It begins with MXFaaS, a mechanism that improves resource utilization by efficiently multiplexing resources during bursts of same-function invocations. Then, it integrates the novel Concord distributed caching system for FaaS environments, and uses SpecFaaS to accelerate end-toend application workflows through speculative service execution. Finally, this thesis improves the energy efficiency of cloud-native environments with two frameworks: EcoFaaS, which uses fine-grained scheduling and dynamic frequency scaling, and SmartOClock, which underprovisions resources and selectively overclocks cores during load spikes.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Stojkovic, Jovan

Contributors dc:contributor

• Torrellas, Josep
• Xu, Tianyin
• Marinov, Darko
• Huang, Jian
• Franke, Hubertus
• Delimitrou, Christina
• Skarlatos, Dimitrios
• Hughes, Christopher J.

• Cloud Computing
• Serverless Computing
• Microservices
• Computer Architecture

Statement dc:rights

• Copyright 2025 Jovan Stojkovic

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/130086

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F130086