Improving speculative retrieval-augmented generation via verifier scoring

The rapid progress of large language models (LLMs) has revolutionized various natural language processing (NLP) tasks, including text generation, question answering, and summarization. However, the computational cost and latency associated with LLMs remain significant challenges, particularly in real-time applications. What's more, the heavy fine-tuning cost for each downstream task is also unbearable in the preparation phase. To address these issues, researchers have explored various techniques to accelerate both inference and fine-tuning without compromising the quality of the generated text. One of the recent breakthroughs is Speculative Decoding. It leverages a smaller, faster "draft" model to propose multiple candidate drafts that are then verified in parallel by a larger, more generative, and more accurate "verification" model. This method significantly reduces the number of calls to the larger model, thereby speeding up the inference process. Also, the drafter model is much easier to fine-tune than larger models. In this thesis, we are inspired by an innovative extension of the speculative decoding framework by integrating Retrieval-Augmented Generation (RAG). RAG enhances the capabilities of language models by retrieving relevant documents from a knowledge base to inform the generation process. Our research represents a comprehensive analysis Speculative RAG, aims to further optimize the inference pipeline by categorizing the retrieved documents and feeding them to a smaller drafter model, which generates multiple draft continuations in parallel. These drafts are then evaluated by a larger, general-purpose verifier model to select the most accurate and contextually appropriate continuation.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Chen, Shixin

Contributors dc:contributor

• Kindratenko, Volodymyr

• Speculative Decoding
• RAG
• LLMs

Statement dc:rights

• Copyright 2025 Shixin Chen

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129272

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129272