An end-to-end benchmarking framework for retrieval-augmented generation systems

As Large Language Models (LLMs) transition from experimental prototypes to production-grade services, Retrieval-Augmented Generation (RAG) has emerged as the de facto paradigm for mitigating hallucinations and incorporating up-to-date knowledge. While the accuracy of RAG systems has been extensively studied, the system performance—specifically regarding throughput, latency, and memory efficiency at scale—remains largely unexplored. However, current RAG benchmarking efforts are predominantly accuracy-centric, focusing on metrics like precision and recall while neglecting the implications of the underlying retrieval infrastructure and generation bottlenecks. Consequently, developers face significant challenges in navigating the complex trade-offs between vector database configurations, retrieval strategies, and generative model parameters. This thesis presents a RAG-based AI system benchmarking framework (RASB) for characterizing the system performance of RAG pipelines. To enable a holistic evaluation, RASB decouples the RAG workflow into modular components—embedding, indexing, retrieval, and generation—allowing for fine-grained analysis of each stage. We rethink the evaluation methodology by shifting the focus from pure answer quality to system efficiency, exploring multiple dimensions such as varying batch sizes, vector database index types, and embedding dimensions. RASB provides a testbed that supports modular RAG pipelines with major vector databases and LLM backends, automating the collection of performance metrics that include end-to-end throughput, GPU memory consumption, and context recall. To evaluate diverse usage scenarios, RASB integrates a configurable workload generator that drives experiments using both real-world and synthetic datasets. We demonstrate RASB’s capability through a comprehensive set of experiments conducted on popular Vector Databases and LLM backends.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Xu, Yuan

Contributors dc:contributor

• Huang, Jian

• Retrieval-Augmented Generation (RAG)
• Benchmarking framework
• AI system performance

Statement dc:rights

• Copyright 2025 Yuan Xu

Language dc:language

en

Handle dc:identifier

https://hdl.handle.net/2142/132602

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/132602