Discovering, autogenerating, and evaluating distractors for Parsons problems in CS1

This dissertation presents work on evaluating the use of distractor blocks in Parsons problems, a problem type aimed at supporting novice programmers in learning to write code. Since their introduction by Parsons & Hayden [1], distractors have become common place in studies investigating Parsons problems. However, prior to the work presented in this dissertation, there had been a limited number of studies investigating their utility. In filling this gap, my work has focused on investigating methods of developing distractors and the impact of distractors in both summative and formative contexts. I first introduce a method of creating distractor templates from analysis of code writing errors and using those templates to support the autogeneration of distractors [2], a process that has been integrated into CodeSpec, a programming practice and eTextbook platform [3]. I present a series of studies comparing Parsons problems on exams that include distractors to those that do not. These studies have also included a comparison between different methods of including distractors, specifically comparing the use of distractors that are jumbled among the other options to distractors that are visually-grouped with their correct alternative [4, 5]. Additionally, I present a study comparing the impact on learning gains when practicing with Parsons problems that include distractors versus those without distractors. Learning gains are measured through pre- and post-assessment performance. In support of this work, I have also added the ability to associate feedback with distractors in the PrairieLearn assessment platform. The findings presented suggest that in summative contexts the presence of a single distractor group does not meaningfully increase question difficulty or item quality while causing students to spend significantly more time on the problems, a concern in a time constrained context such as an exam. However, as more distractor groups are added difficulty continues to increase with minimal impact—either positively or negatively—to item quality. This suggests that when distractors are added in sufficient quantities they can be useful for reducing the ceiling effect on score we observed in Parsons problems used in this study that do not include distractors. In formative contexts, the findings I present suggest that the presence of distractors causes students to engage more deeply with the code presented in blocks they are grouped with. The results of a randomized controlled trial indicate that students practicing with questions including distractors had measurable learning gains as measured through the post-test.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Smith IV, David Hamilton

Contributors dc:contributor

• Zilles, Craig
• Lewis, Colleen
• Ericson, Barbara
• Cunningham, Kathryn

• Computing Education
• Parsons problems
• CS1
• Distractors
• Item Response Theory
• Classical Test Theory
• Measurement Theory

Statement dc:rights

• Copyright 2025 David Smith IV

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129414

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129414