Disdoc: AI Teaching Assistant for Computer Science Courses

As enrollment in Computer Science grows, traditional help-seeking opportunities for students, such as office hours and forums, become less effective due to rising student-to-teaching assistant ratios. To address this issue, research has investigated large language models (LLMs) to provide individualized help to students at scale. However, prior research primarily targets introductory computing courses, does not fully connect LLMs to course material, and does not expose relevant course material to students. As a result, existing approaches do not adapt well to advanced computing courses and limit opportunities for students to develop self-sufficiency. To address this, we present Disdoc, an LLM-based question and answer tool for students in advanced computing courses. Disdoc presents snippets of course material relevant to student questions and generates answers using an LLM. To include course-specific information in answers, we connect the LLM to all course material through retrieval-augmented generation (RAG). To ensure the RAG system retrieves the most relevant information, we organize course material into question categories. We evaluated Disdoc in a research study on a 340-student Computer Systems class at Virginia Tech, where we tracked student reviews, activity, and exit survey responses. Students indicated that Disdoc was helpful, particularly for questions about course assignments. Usage data revealed that students strongly preferred to see LLM-generated answers and rarely clicked on outgoing links, suggesting they were satisfied with the LLM-generated answers and snippets of relevant course material.

Name thesis:degree_name

Master of Science

Level thesis:degree_level

masters

Discipline thesis:degree_discipline

Computer Science & Applications

Department dc:contributor.department

Computer Science and#38; Applications

Grantor dc:publisher

Virginia Tech

Year dc:date.issued

2025

Author dc:creator

• Doney, Brendan Robert

Chair dc:contributor.committeechair

• Back, Godmar Volker

Committee members dc:contributor.committeemember

• Hamouda, Sally
• Ellis, Margaret O.'Neil

• Large Language Models
• Generative Artificial Intelligence
• AI Teaching Assistant
• Computer Science Education

Statement dc:rights

• Creative Commons Attribution-ShareAlike 4.0 International

Licence dc:rights.uri

• http://creativecommons.org/licenses/by-sa/4.0/

Language dc:language.iso

en

Dc Identifier Other

vt_gsexam:42580

OAI identifier oai:identifier

oai:vtechworks.lib.vt.edu:10919/125119