Evaluating the Effect of Domain-Specific Large Language Models on Question and Response

<p>Large Language Models have emerged to great fanfare in the Information Technology market. Business and Information Technology leaders are currently exploring ways to apply these models to assist their organizations in executing business processes and generating innovation. Software vendors, consultants, and academics promote various approaches to making Large Language Models work effectively for business. However, little academic literature is available today that quantifies the degree of improvement possible with these domain-specific approaches over the standard capabilities of generalized Large Language Models.</p> <p>The study seeks to quantify the benefits of one approach, Retrieval Augmented Generation. The study uses a collection of questions across several topics with known reference answers. The context for these questions is used to assemble a study corpus. Responses are generated by both a standard Large Language Model system and a Retrieval Augmented Generation system. The study analyzes the quality of generated responses to determine the degree to which the Retrieval Augmented Generation responses differ from those generated by the standard Large Language Model.</p>

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Year dc:date.available

2024

Author dc:creator

• Crippen, Phillip D.

Contributors dc:contributor

• Ryan Elmore
• Kellie Keeling
• Benjamin Williams

• Corpus
• Large language model (LLM)
• Non-factoid question taxonomy
• Retrieval augmented generation (RAG)
• Vector database
• Business
• Software Engineering
• Technology and Innovation

Statement dc:rights

• <p>Copyright is held by the author. User is responsible for all copyright compliance.</p>

Language dc:language

English (eng)

Repository record dc:identifier

https://digitalcommons.du.edu/etd/2418

OAI identifier oai:identifier

oai:digitalcommons.du.edu:etd-3410