{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/125524"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/125524","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Strategies and exercises for assessing code writing and reading ability at scale","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-02-04 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-02-04 without embargo terms","abstract_has_math":false,"creators":["Fowler, Max"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Zilles, Craig","Lewis, Colleen","Karahalios, Karrie","Porter, Leo"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024-06-18","date_published":"2024-06-18","updated_at":"2026-07-22T22:25:02Z","subjects":["Computer Science Education","Explain In Plain English","Eipe","Isomorphic Questions","Assessment"],"languages":["en","eng"],"rights":["Copyright 2024 Max Fowler"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/125524","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Zilles, Craig","Lewis, Colleen","Karahalios, Karrie","Porter, Leo"]},{"key":"dc:creator","label":"Author","values":["Fowler, Max"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2024-06-18","2024-08"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computer Science Education","Explain In Plain English","Eipe","Isomorphic Questions","Assessment"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2024 Max Fowler"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/125524"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-02-04 without embargo terms","The student, Max Fowler, accepted the attached license on 2024-06-17 at 12:14.","The student, Max Fowler, submitted this Dissertation for approval on 2024-06-17 at 12:23.","This Dissertation was approved for publication on 2024-06-18 at 12:42.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20844 on 2025-02-04 at 21:03:44","Fair and effective assessment at scale necessitates large numbers of relevant exercises. In computer science, the need for a multitude of exercises is further complicated by the nature of the skills we assess. The tightly coupled nature of foundational code-related skills, such as tracing, code reading, and code writing - requires a range of question types in order to properly practice and assess programming skills. In this dissertation, I present work on two strategies for effective programming skill assessment at scale: a strategy for rapidly authoring isomorphic programming questions and multiple techniques for autograding Explain in Plain English (EiPE) exercises. Small, auto-gradable programming exercises provide a useful tool with which to assess students' code writing ability. This dissertation presents multiple semesters of using a surface feature permutation strategy to generate isomorphic programming exercises from existing questions. In particular, this strategy is designed to produce item isomorphs, questions for which the underlying solution strategy does not change significantly. This strategy ranges from changing simple features, such as function names, to more complicated changes such as changing exercise-relevant constants or data types. With multiple semesters of data from a non-major introductory Python course and linear regression, we show evidence that the strategy can be effective. When comparing original questions and their permutations, the latter have at most a 5 -- 11 percentage point difference in score. Of the set of possible features, the more \"complex\" features may produce questions that are harder than is ideal to be considered isomorphic questions. EiPE exercises present students with pieces of code and ask them to write a high-level description of the code's purpose. This dissertation presents the result of interviews with faculty who use EiPE exercises with respect to how they may grade such exercises and what they are useful for. Following from these interviews, the likelihood that some general grading standard could be applied is raised, with a focus on answers being suitably high-level, technically accurate, and unambiguous. As the grading of EiPE questions can be time intensive for large classes, we also present multiple approaches to autograding EiPE exercises. Two of these methods are based on simple linear classifiers with expert labeled student responses and two are based on using large language models (LLMs) in the grading pipeline. We find the majority of these autograders have similar performance in the 86 -- 88% range versus our expert established ground truth. Further, we find this performance is comparable to the accuracy of the average course teaching assistant (TA). Together, this dissertation presents two pathways to produce a large number of reliable questions for large scale assessment in order to assess code writing and code reading ability."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Strategies and exercises for assessing code writing and reading ability at scale"]}]}],"canonical_facts":{"dc:contributor":["Zilles, Craig","Lewis, Colleen","Karahalios, Karrie","Porter, Leo"],"dc:creator":["Fowler, Max"],"dc:date":["2024-06-18","2024-08"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-02-04 without embargo terms","The student, Max Fowler, accepted the attached license on 2024-06-17 at 12:14.","The student, Max Fowler, submitted this Dissertation for approval on 2024-06-17 at 12:23.","This Dissertation was approved for publication on 2024-06-18 at 12:42.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20844 on 2025-02-04 at 21:03:44","Fair and effective assessment at scale necessitates large numbers of relevant exercises. In computer science, the need for a multitude of exercises is further complicated by the nature of the skills we assess. The tightly coupled nature of foundational code-related skills, such as tracing, code reading, and code writing - requires a range of question types in order to properly practice and assess programming skills. In this dissertation, I present work on two strategies for effective programming skill assessment at scale: a strategy for rapidly authoring isomorphic programming questions and multiple techniques for autograding Explain in Plain English (EiPE) exercises. Small, auto-gradable programming exercises provide a useful tool with which to assess students' code writing ability. This dissertation presents multiple semesters of using a surface feature permutation strategy to generate isomorphic programming exercises from existing questions. In particular, this strategy is designed to produce item isomorphs, questions for which the underlying solution strategy does not change significantly. This strategy ranges from changing simple features, such as function names, to more complicated changes such as changing exercise-relevant constants or data types. With multiple semesters of data from a non-major introductory Python course and linear regression, we show evidence that the strategy can be effective. When comparing original questions and their permutations, the latter have at most a 5 -- 11 percentage point difference in score. Of the set of possible features, the more \"complex\" features may produce questions that are harder than is ideal to be considered isomorphic questions. EiPE exercises present students with pieces of code and ask them to write a high-level description of the code's purpose. This dissertation presents the result of interviews with faculty who use EiPE exercises with respect to how they may grade such exercises and what they are useful for. Following from these interviews, the likelihood that some general grading standard could be applied is raised, with a focus on answers being suitably high-level, technically accurate, and unambiguous. As the grading of EiPE questions can be time intensive for large classes, we also present multiple approaches to autograding EiPE exercises. Two of these methods are based on simple linear classifiers with expert labeled student responses and two are based on using large language models (LLMs) in the grading pipeline. We find the majority of these autograders have similar performance in the 86 -- 88% range versus our expert established ground truth. Further, we find this performance is comparable to the accuracy of the average course teaching assistant (TA). Together, this dissertation presents two pathways to produce a large number of reliable questions for large scale assessment in order to assess code writing and code reading ability."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/125524"],"dc:language":["en","eng"],"dc:rights":["Copyright 2024 Max Fowler"],"dc:subject":["Computer Science Education","Explain In Plain English","Eipe","Isomorphic Questions","Assessment"],"dc:title":["Strategies and exercises for assessing code writing and reading ability at scale"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:02Z"}