Improving Question Answering Systems with Retrieval Augmented Generation

Large language models (LLMs) have been proven to be state-of-the-art solutions for many NLP benchmarks. However, LLMs in real applications face many limitations. Although such models are seen to contain real-world knowledge, it is kept implicitly in their parameters that cannot be revised and extended unless expensive additional training is performed. These models can hallucinate by confidently producing human-like texts which might contain misleading information. The knowledge limitation and the tendency to hallucinate cause LLMs to struggle with out-of-domain settings. Furthermore, LLMs lack transparency in that their responses are products of big black-box models. While fine-tuning can mitigate some of these issues, it requires high computing resources. On the other hand, retrieval augmentation has been used to tackle knowledge-intensive tasks and proven by recent studies to be effective when coupled with LLMs. In this thesis, we explore Retrieval-Augmented Generation (RAG), a framework to augment generative LLMs with a neural retriever component, in a domain-specific question answering (QA) task. Empirically, we study how RAG helps LLMs in knowledge-intensive situations and explore design decisions in building a RAG pipeline. Our findings underscore the benefits of RAG in the studied situation by showing that leveraging retrieval augmentation yields significant improvement on QA performance over using a pre-trained LLM alone. Furthermore, incorporating RAG in an LLM-driven QA pipeline results in a QA system that accompanies its predictions with evidence documents, leading to a more trustworthy and grounded AI applications.

Grantor dc:publisher

Helsingin yliopisto

Year dc:date.issued

2024

Author dc:creator

• Trangcasanchai, Sathianpong

Contributors dc:contributor

• Helsingin yliopisto, Matemaattis-luonnontieteellinen tiedekunta
• University of Helsinki, Faculty of Science
• Helsingfors universitet, Matematisk-naturvetenskapliga fakulteten

Language dc:language.iso

eng

Identifier URI

URN:NBN:fi:hulib-202406253316

OAI identifier oai:identifier

oai:helda.helsinki.fi:10138/577810