Pushing the limits of long context LLM inference via KV cache compression

Efficiently deploying/serving LLMs has become remarkably challenging due to their excessive memory and computational requirements. A critical bottleneck in LLM inference is the memory footprint of the Key-Value (KV) cache, particularly in tasks involving long-context understanding and generation. To address these challenges we introduce MiniKV, a hybrid KV cache optimization technique which compresses the KV cache by combining token eviction and 2-bit quantization. Our approach aims to significantly reduce memory usage while maintaining high accuracy on downstream tasks such as question answering, summarization, code generation, and retrieval. Our evaluations demonstrate that MiniKV achieves an 86% reduction in KV cache size while recovering over 98.5% accuracy across downstream tasks. This sets a new state-of-the-art in balancing accuracy and compression, with notable improvements in inference latency and throughput.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Sharma, Akshat

Contributors dc:contributor

• Zhang, Minjia

• Systems for LLM
• LLM Inference

Statement dc:rights

• Copyright 2025 Akshat Sharma

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129176

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129176