Lessons learnt developing and deploying grading mechanisms for EiPE code-reading questions in CS1 classes

Previous research has identified that the ability to understand the high-level purpose of a piece of code is an important developmental skill that is harder to master than executing the same piece of code in one’s head for a given input (“code tracing”), but easier to master than writing the code. One way to help students to practice this skill in the middle ground is by asking them Explain in Plain English (EiPE) questions, where they are asked to explain the purpose of a piece of code at a high level. Prior works involving EiPE questions have used scoring rubrics that do not adequately handle the three dimensions of answer quality: correctness, level of abstraction, and ambiguity. Also, these studies have been carried out in limited experimental settings with manual grading, which did not shed light on how EiPE questions can be deployed in real world classrooms without overwhelming grading workload. In this work, we cover both unaddressed issues. First, we describe our efforts in validating a 7-point rubric that the research group has developed for scoring student responses to EiPE questions. Second, we describe the deployment of an imperfect NLP-based automatic grading system for these EiPE responses on an exam in CS105, a large-enrollment CS1 course at the University of Illinois, finding that the auto-grader has an accuracy similar to that of TA’s who teach the course. We study allowing students to attempt an EiPE question multiple times (without penalty based on the number of attempts used) in exam settings as a strategy to mitigate potential student dissatisfaction due to the imperfect grading system mistakenly rejecting a correct answer. We also characterize common student errors, auto-grader failure, and discuss the lessons learnt in this process.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2020

Author dc:creator

• Azad, Sushmita

Contributors dc:contributor

• Zilles, Craig

• CSEducation
• CS1
• Code-reading
• EiPE

Statement dc:rights

• Copyright 2020 Sushmita Azad

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/108199

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/108199