Design and implementation of learning-based storage systems: a holistic approach

Storage systems have evolved over decades into a complicated ecosystem that includes storage hardware, system software, storage infrastructure, and data applications. This growing complexity poses significant challenges for storage development and deployment. Traditional human-driven, heuristic-based approaches to building storage systems cannot rapidly meet the ever-increasing demands for storage performance and efficiency. As we embrace the advancement of machine learning (ML), it is the golden age today to invent new approaches to building storage systems. This dissertation focuses on the design and implementation of learning-based storage systems across the entire storage stack, which develops learning-based approaches to optimize storage performance, resource efficiency, and management. The storage performance of traditional hardware and software cannot keep up with the increasing demands of applications. For example, in solid-state drives (SSDs), the flash translation layer (FTL) manages performance-critical metadata structures using human-driven heuristics, but it fails to adapt to different workload patterns and results in severe performance loss. To address this performance challenge, this dissertation proposes the first learning-based flash translation layer, LeaFTL, that can dynamically capture data access patterns of storage workloads at runtime. It significantly reduces the memory footprint of the address mapping table by grouping a large set of mapping entries into a learned segment. The saved memory space further benefits data caching and improves the overall storage performance. Storage resource efficiency is also critical as it directly affects the operational costs of storage infrastructure. For instance, cloud platforms manage storage resources at scale and it is beneficial for them to achieve high storage utilization. However, our study reveals that storage resources in modern cloud platforms are severely underutilized. To address this resource efficiency challenge, this dissertation proposes a learning-based storage harvesting framework named BlockFlex, that can dynamically harvest idle storage capacity and bandwidth to improve cloud storage utilization. BlockFlex leverages a lightweight online learning approach to predict resource utilization and storage resource demands, based on which it enables accurate storage resource harvesting. Storage management is becoming complicated with the rapid development of the software and hardware over the past decades. This is especially true for cloud platforms, where the cloud storage resource is shared by multiple tenants with complex storage states and dynamic workload characteristics. Cloud platform enforces isolation mechanisms as it manages collocated tenants, but weak isolation incurs high performance interference and strong isolation causes low storage utilization. This dissertation explores reinforcement learning (RL) techniques to combat this fundamental tussle, leveraging its unique advantages of optimizing decisions in the complex and dynamic environment. It employs multi-agent reinforcement learning to dynamically manage resource scheduling across multiple tenants. Our experiments show that it can achieve both high storage utilization and performance isolation.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Sun, Jinghan

Contributors dc:contributor

• Huang, Jian
• Snir, Marc
• Gropp, William
• Patel, Sanjay
• Swift, Michael
• Chang, Jichuan

• Storage Systems
• Machine Learning For Systems
• Hardware Software Co-design
• Solid-state Drive
• Cloud Storage

Statement dc:rights

• Copyright 2025 Jinghan Sun

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129928

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129928