{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/108199"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/108199","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Lessons learnt developing and deploying grading mechanisms for EiPE code-reading questions in CS1 classes","abstract":"Previous research has identified that the ability to understand the high-level purpose of a piece of code is an important developmental skill that is harder to master than executing the same piece of code in one’s head for a given input (“code tracing”), but easier to master than writing the code. One way to help students to practice this skill in the middle ground is by asking them Explain in Plain English (EiPE) questions, where they are asked to explain the purpose of a piece of code at a high level. Prior works involving EiPE questions have used scoring rubrics that do not adequately handle the three dimensions of answer quality: correctness, level of abstraction, and ambiguity. Also, these studies have been carried out in limited experimental settings with manual grading, which did not shed light on how EiPE questions can be deployed in real world classrooms without overwhelming grading workload. In this work, we cover both unaddressed issues. First, we describe our efforts in validating a 7-point rubric that the research group has developed for scoring student responses to EiPE questions. Second, we describe the deployment of an imperfect NLP-based automatic grading system for these EiPE responses on an exam in CS105, a large-enrollment CS1 course at the University of Illinois, finding that the auto-grader has an accuracy similar to that of TA’s who teach the course. We study allowing students to attempt an EiPE question multiple times (without penalty based on the number of attempts used) in exam settings as a strategy to mitigate potential student dissatisfaction due to the imperfect grading system mistakenly rejecting a correct answer. We also characterize common student errors, auto-grader failure, and discuss the lessons learnt in this process.","abstract_html":"Previous research has identified that the ability to understand the high-level purpose of a piece of code is an important developmental skill that is harder to master than executing the same piece of code in one’s head for a given input (“code tracing”), but easier to master than writing the code. One way to help students to practice this skill in the middle ground is by asking them Explain in Plain English (EiPE) questions, where they are asked to explain the purpose of a piece of code at a high level. Prior works involving EiPE questions have used scoring rubrics that do not adequately handle the three dimensions of answer quality: correctness, level of abstraction, and ambiguity. Also, these studies have been carried out in limited experimental settings with manual grading, which did not shed light on how EiPE questions can be deployed in real world classrooms without overwhelming grading workload. In this work, we cover both unaddressed issues. First, we describe our efforts in validating a 7-point rubric that the research group has developed for scoring student responses to EiPE questions. Second, we describe the deployment of an imperfect NLP-based automatic grading system for these EiPE responses on an exam in CS105, a large-enrollment CS1 course at the University of Illinois, finding that the auto-grader has an accuracy similar to that of TA’s who teach the course. We study allowing students to attempt an EiPE question multiple times (without penalty based on the number of attempts used) in exam settings as a strategy to mitigate potential student dissatisfaction due to the imperfect grading system mistakenly rejecting a correct answer. We also characterize common student errors, auto-grader failure, and discuss the lessons learnt in this process.","abstract_has_math":false,"creators":["Azad, Sushmita"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Zilles, Craig"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2020,"date_issued":"2020-08-26T23:58:50Z","date_published":"2020-08-26T23:58:50Z","updated_at":"2026-07-22T22:24:48Z","subjects":["CSEducation","CS1","Code-reading","EiPE"],"languages":["en"],"rights":["Copyright 2020 Sushmita Azad"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/108199","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Zilles, Craig"]},{"key":"dc:creator","label":"Author","values":["Azad, Sushmita"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2020-08-26T23:58:50Z","2022-08-26T23:58:55Z","2020-05-14","2020-05"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["CSEducation","CS1","Code-reading","EiPE"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2020 Sushmita Azad"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/108199"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Previous research has identified that the ability to understand the high-level purpose of a piece of code is an important developmental skill that is harder to master than executing the same piece of code in one’s head for a given input (“code tracing”), but easier to master than writing the code. One way to help students to practice this skill in the middle ground is by asking them Explain in Plain English (EiPE) questions, where they are asked to explain the purpose of a piece of code at a high level. Prior works involving EiPE questions have used scoring rubrics that do not adequately handle the three dimensions of answer quality: correctness, level of abstraction, and ambiguity. Also, these studies have been carried out in limited experimental settings with manual grading, which did not shed light on how EiPE questions can be deployed in real world classrooms without overwhelming grading workload. In this work, we cover both unaddressed issues. First, we describe our efforts in validating a 7-point rubric that the research group has developed for scoring student responses to EiPE questions. Second, we describe the deployment of an imperfect NLP-based automatic grading system for these EiPE responses on an exam in CS105, a large-enrollment CS1 course at the University of Illinois, finding that the auto-grader has an accuracy similar to that of TA’s who teach the course. We study allowing students to attempt an EiPE question multiple times (without penalty based on the number of attempts used) in exam settings as a strategy to mitigate potential student dissatisfaction due to the imperfect grading system mistakenly rejecting a correct answer. We also characterize common student errors, auto-grader failure, and discuss the lessons learnt in this process.","Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2022-05-01","The student, Sushmita Azad, accepted the attached license on 2020-05-13 at 11:51.","The student, Sushmita Azad, submitted this Thesis for approval on 2020-05-13 at 12:01.","This Thesis was approved for publication on 2020-05-14 at 16:37.","DSpace SAF Submission Ingestion Package generated from Vireo submission #15379 on 2020-08-25 at 17:31:24","Made available in DSpace on 2020-08-26T23:58:50Z (GMT). No. of bitstreams: 2 AZAD-THESIS-2020.pdf: 936626 bytes, checksum: 13beb21560b89e0c19c02d345e479358 (MD5) LICENSE.txt: 4210 bytes, checksum: 647f7c4952ee3081197cd56f1c2cb6c2 (MD5) Previous issue date: 2020-05-14","Embargo set by: Seth Robbins for item 115812 Lift date: 2022-08-26T23:58:55Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","U of I Only"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Lessons learnt developing and deploying grading mechanisms for EiPE code-reading questions in CS1 classes"]}]}],"canonical_facts":{"dc:contributor":["Zilles, Craig"],"dc:creator":["Azad, Sushmita"],"dc:date":["2020-08-26T23:58:50Z","2022-08-26T23:58:55Z","2020-05-14","2020-05"],"dc:description":["Previous research has identified that the ability to understand the high-level purpose of a piece of code is an important developmental skill that is harder to master than executing the same piece of code in one’s head for a given input (“code tracing”), but easier to master than writing the code. One way to help students to practice this skill in the middle ground is by asking them Explain in Plain English (EiPE) questions, where they are asked to explain the purpose of a piece of code at a high level. Prior works involving EiPE questions have used scoring rubrics that do not adequately handle the three dimensions of answer quality: correctness, level of abstraction, and ambiguity. Also, these studies have been carried out in limited experimental settings with manual grading, which did not shed light on how EiPE questions can be deployed in real world classrooms without overwhelming grading workload. In this work, we cover both unaddressed issues. First, we describe our efforts in validating a 7-point rubric that the research group has developed for scoring student responses to EiPE questions. Second, we describe the deployment of an imperfect NLP-based automatic grading system for these EiPE responses on an exam in CS105, a large-enrollment CS1 course at the University of Illinois, finding that the auto-grader has an accuracy similar to that of TA’s who teach the course. We study allowing students to attempt an EiPE question multiple times (without penalty based on the number of attempts used) in exam settings as a strategy to mitigate potential student dissatisfaction due to the imperfect grading system mistakenly rejecting a correct answer. We also characterize common student errors, auto-grader failure, and discuss the lessons learnt in this process.","Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2022-05-01","The student, Sushmita Azad, accepted the attached license on 2020-05-13 at 11:51.","The student, Sushmita Azad, submitted this Thesis for approval on 2020-05-13 at 12:01.","This Thesis was approved for publication on 2020-05-14 at 16:37.","DSpace SAF Submission Ingestion Package generated from Vireo submission #15379 on 2020-08-25 at 17:31:24","Made available in DSpace on 2020-08-26T23:58:50Z (GMT). No. of bitstreams: 2 AZAD-THESIS-2020.pdf: 936626 bytes, checksum: 13beb21560b89e0c19c02d345e479358 (MD5) LICENSE.txt: 4210 bytes, checksum: 647f7c4952ee3081197cd56f1c2cb6c2 (MD5) Previous issue date: 2020-05-14","Embargo set by: Seth Robbins for item 115812 Lift date: 2022-08-26T23:58:55Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","U of I Only"],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/108199"],"dc:language":["en"],"dc:rights":["Copyright 2020 Sushmita Azad"],"dc:subject":["CSEducation","CS1","Code-reading","EiPE"],"dc:title":["Lessons learnt developing and deploying grading mechanisms for EiPE code-reading questions in CS1 classes"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:48Z"}