{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/105768"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/105768","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Use of symbolic execution as automated grading tool for introductory programming courses","abstract":"The use of automated grading tools to provide feedback to students is common in Computer Science education. The first step of automated grading is to find defects in the student program. However, finding bugs in code has never been easy. Current automated grading tools do not focus on identifying defects inside student code. Instead, the most common way to determine correctness is to compare computation results for a fixed set of test cases. It takes time and effort to design a set of test cases that test the student code thoroughly. In practice, the tests used for grading are often insufficient for accurate diagnosis. Meanwhile, automated testing tools for commercial code have been developing for some time. Using these tools, we can deliver scalable, high-quality feedback. We believe that with some modifications, automated testing tools for commercial code can also improve Computer Science education. In this thesis, we present our modification and utilization of automated testing on student assignments in an introductory programming course. We implemented a framework to collect student codes and to apply industrial automated testing to their codes. To enable scalable feedback generation by the tools, we integrated I/O functions into our tool, developed an algorithm to speed up execution through loops and implemented a cross-execution cache for queries. We also implemented interpretation of the results of the tools in a way that students can understand easily. In order to provide timely feedback to students, we modified the current state-of-the-art symbolic execution tool KLEE. Because the I/O functions are used heavily by students, we integrated them into our tool. We implemented a new path exploration algorithm that reduces the number of paths needed to achieve high coverage. We have tested our Loop Reduction algorithm with 235 student-generated versions of a classroom assignment. We demonstrate how our algorithm helps to achieve the same coverage with speedup of 11.8X for 117 out of the 235 programs, while adding 15% max observed and 2% average overhead over the 50% of programs not benefiting from the technique. The maximum speedup for a single program is 52.3X. We also implemented a cross-execution cache for the SMT queries. We found that the transmission and translation of query creates an overhead of about 20% coupled with the fact that query solving time only takes about 60% of overall analysis time. We find that we can achieve a speedup of 1.5x. We deployed our framework on eight different introductory C programming assignments here at the University of Illinois at Urbana-Champaign for over four years. The setup work takes a senior undergraduate student about 5-10 hours for each assignment. The results show that the automated feedback generation framework can detect more bugs inside student programs, as well as errors in the assignment specification, released tests, and released gold program. We can generate helpful feedback to students that helps them to learn from their own mistakes, and help instructors to identify misunderstood concepts. We can also grade more thoroughly and more fairly. We believe that automated testing tools based on symbolic execution are the future of improving Computer Science education.","abstract_html":"The use of automated grading tools to provide feedback to students is common in Computer Science education. The first step of automated grading is to find defects in the student program. However, finding bugs in code has never been easy. Current automated grading tools do not focus on identifying defects inside student code. Instead, the most common way to determine correctness is to compare computation results for a fixed set of test cases. It takes time and effort to design a set of test cases that test the student code thoroughly. In practice, the tests used for grading are often insufficient for accurate diagnosis. Meanwhile, automated testing tools for commercial code have been developing for some time. Using these tools, we can deliver scalable, high-quality feedback. We believe that with some modifications, automated testing tools for commercial code can also improve Computer Science education. In this thesis, we present our modification and utilization of automated testing on student assignments in an introductory programming course. We implemented a framework to collect student codes and to apply industrial automated testing to their codes. To enable scalable feedback generation by the tools, we integrated I/O functions into our tool, developed an algorithm to speed up execution through loops and implemented a cross-execution cache for queries. We also implemented interpretation of the results of the tools in a way that students can understand easily. In order to provide timely feedback to students, we modified the current state-of-the-art symbolic execution tool KLEE. Because the I/O functions are used heavily by students, we integrated them into our tool. We implemented a new path exploration algorithm that reduces the number of paths needed to achieve high coverage. We have tested our Loop Reduction algorithm with 235 student-generated versions of a classroom assignment. We demonstrate how our algorithm helps to achieve the same coverage with speedup of 11.8X for 117 out of the 235 programs, while adding 15% max observed and 2% average overhead over the 50% of programs not benefiting from the technique. The maximum speedup for a single program is 52.3X. We also implemented a cross-execution cache for the SMT queries. We found that the transmission and translation of query creates an overhead of about 20% coupled with the fact that query solving time only takes about 60% of overall analysis time. We find that we can achieve a speedup of 1.5x. We deployed our framework on eight different introductory C programming assignments here at the University of Illinois at Urbana-Champaign for over four years. The setup work takes a senior undergraduate student about 5-10 hours for each assignment. The results show that the automated feedback generation framework can detect more bugs inside student programs, as well as errors in the assignment specification, released tests, and released gold program. We can generate helpful feedback to students that helps them to learn from their own mistakes, and help instructors to identify misunderstood concepts. We can also grade more thoroughly and more fairly. We believe that automated testing tools based on symbolic execution are the future of improving Computer Science education.","abstract_has_math":false,"creators":["Gao, Jianxiong"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Electrical & Computer Engr","degree_department":null,"school":null,"contributors":["Lumetta, Steven Sam","Hwu, Wen-mei","Marinov, Darko","Mitra, Sayan"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2019,"date_issued":"2019-11-26T20:49:17Z","date_published":"2019-11-26T20:49:17Z","updated_at":"2026-07-22T22:24:45Z","subjects":["Symbolic Execution Computer Science Education Feedback Program Analysis"],"languages":["en"],"rights":["Copyright 2019 Jianxiong Gao"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/105768","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Lumetta, Steven Sam","Hwu, Wen-mei","Marinov, Darko","Mitra, Sayan"]},{"key":"dc:creator","label":"Author","values":["Gao, Jianxiong"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2019-11-26T20:49:17Z","2021-11-27T10:15:33Z","2019-06-28","2019-08"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Electrical & Computer Engr"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Symbolic Execution Computer Science Education Feedback Program Analysis"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2019 Jianxiong Gao"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/105768"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["The use of automated grading tools to provide feedback to students is common in Computer Science education. The first step of automated grading is to find defects in the student program. However, finding bugs in code has never been easy. Current automated grading tools do not focus on identifying defects inside student code. Instead, the most common way to determine correctness is to compare computation results for a fixed set of test cases. It takes time and effort to design a set of test cases that test the student code thoroughly. In practice, the tests used for grading are often insufficient for accurate diagnosis. Meanwhile, automated testing tools for commercial code have been developing for some time. Using these tools, we can deliver scalable, high-quality feedback. We believe that with some modifications, automated testing tools for commercial code can also improve Computer Science education. In this thesis, we present our modification and utilization of automated testing on student assignments in an introductory programming course. We implemented a framework to collect student codes and to apply industrial automated testing to their codes. To enable scalable feedback generation by the tools, we integrated I/O functions into our tool, developed an algorithm to speed up execution through loops and implemented a cross-execution cache for queries. We also implemented interpretation of the results of the tools in a way that students can understand easily. In order to provide timely feedback to students, we modified the current state-of-the-art symbolic execution tool KLEE. Because the I/O functions are used heavily by students, we integrated them into our tool. We implemented a new path exploration algorithm that reduces the number of paths needed to achieve high coverage. We have tested our Loop Reduction algorithm with 235 student-generated versions of a classroom assignment. We demonstrate how our algorithm helps to achieve the same coverage with speedup of 11.8X for 117 out of the 235 programs, while adding 15% max observed and 2% average overhead over the 50% of programs not benefiting from the technique. The maximum speedup for a single program is 52.3X. We also implemented a cross-execution cache for the SMT queries. We found that the transmission and translation of query creates an overhead of about 20% coupled with the fact that query solving time only takes about 60% of overall analysis time. We find that we can achieve a speedup of 1.5x. We deployed our framework on eight different introductory C programming assignments here at the University of Illinois at Urbana-Champaign for over four years. The setup work takes a senior undergraduate student about 5-10 hours for each assignment. The results show that the automated feedback generation framework can detect more bugs inside student programs, as well as errors in the assignment specification, released tests, and released gold program. We can generate helpful feedback to students that helps them to learn from their own mistakes, and help instructors to identify misunderstood concepts. We can also grade more thoroughly and more fairly. We believe that automated testing tools based on symbolic execution are the future of improving Computer Science education.","Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2021-08-01","The student, Jianxiong Gao, accepted the attached license on 2019-06-27 at 14:34.","The student, Jianxiong Gao, submitted this Dissertation for approval on 2019-06-27 at 14:45.","This Dissertation was approved for publication on 2019-06-28 at 13:18.","DSpace SAF Submission Ingestion Package generated from Vireo submission #14098 on 2019-11-26 at 13:03:57","Made available in DSpace on 2019-11-26T20:49:17Z (GMT). No. of bitstreams: 2 GAO-DISSERTATION-2019.pdf: 1350849 bytes, checksum: a2affd9c03e2b44cfa8edd4fa9fd932a (MD5) LICENSE.txt: 4210 bytes, checksum: 611de712044ab8171a28d2e107f784e0 (MD5) Previous issue date: 2019-06-28","Embargo set by: Seth Robbins for item 112913 Lift date: 2021-11-26T20:49:41Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","U of I Only Restriction Lifted for Item 112913 on 2021-11-27T10:15:33Z."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Use of symbolic execution as automated grading tool for introductory programming courses"]}]}],"canonical_facts":{"dc:contributor":["Lumetta, Steven Sam","Hwu, Wen-mei","Marinov, Darko","Mitra, Sayan"],"dc:creator":["Gao, Jianxiong"],"dc:date":["2019-11-26T20:49:17Z","2021-11-27T10:15:33Z","2019-06-28","2019-08"],"dc:description":["The use of automated grading tools to provide feedback to students is common in Computer Science education. The first step of automated grading is to find defects in the student program. However, finding bugs in code has never been easy. Current automated grading tools do not focus on identifying defects inside student code. Instead, the most common way to determine correctness is to compare computation results for a fixed set of test cases. It takes time and effort to design a set of test cases that test the student code thoroughly. In practice, the tests used for grading are often insufficient for accurate diagnosis. Meanwhile, automated testing tools for commercial code have been developing for some time. Using these tools, we can deliver scalable, high-quality feedback. We believe that with some modifications, automated testing tools for commercial code can also improve Computer Science education. In this thesis, we present our modification and utilization of automated testing on student assignments in an introductory programming course. We implemented a framework to collect student codes and to apply industrial automated testing to their codes. To enable scalable feedback generation by the tools, we integrated I/O functions into our tool, developed an algorithm to speed up execution through loops and implemented a cross-execution cache for queries. We also implemented interpretation of the results of the tools in a way that students can understand easily. In order to provide timely feedback to students, we modified the current state-of-the-art symbolic execution tool KLEE. Because the I/O functions are used heavily by students, we integrated them into our tool. We implemented a new path exploration algorithm that reduces the number of paths needed to achieve high coverage. We have tested our Loop Reduction algorithm with 235 student-generated versions of a classroom assignment. We demonstrate how our algorithm helps to achieve the same coverage with speedup of 11.8X for 117 out of the 235 programs, while adding 15% max observed and 2% average overhead over the 50% of programs not benefiting from the technique. The maximum speedup for a single program is 52.3X. We also implemented a cross-execution cache for the SMT queries. We found that the transmission and translation of query creates an overhead of about 20% coupled with the fact that query solving time only takes about 60% of overall analysis time. We find that we can achieve a speedup of 1.5x. We deployed our framework on eight different introductory C programming assignments here at the University of Illinois at Urbana-Champaign for over four years. The setup work takes a senior undergraduate student about 5-10 hours for each assignment. The results show that the automated feedback generation framework can detect more bugs inside student programs, as well as errors in the assignment specification, released tests, and released gold program. We can generate helpful feedback to students that helps them to learn from their own mistakes, and help instructors to identify misunderstood concepts. We can also grade more thoroughly and more fairly. We believe that automated testing tools based on symbolic execution are the future of improving Computer Science education.","Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2021-08-01","The student, Jianxiong Gao, accepted the attached license on 2019-06-27 at 14:34.","The student, Jianxiong Gao, submitted this Dissertation for approval on 2019-06-27 at 14:45.","This Dissertation was approved for publication on 2019-06-28 at 13:18.","DSpace SAF Submission Ingestion Package generated from Vireo submission #14098 on 2019-11-26 at 13:03:57","Made available in DSpace on 2019-11-26T20:49:17Z (GMT). No. of bitstreams: 2 GAO-DISSERTATION-2019.pdf: 1350849 bytes, checksum: a2affd9c03e2b44cfa8edd4fa9fd932a (MD5) LICENSE.txt: 4210 bytes, checksum: 611de712044ab8171a28d2e107f784e0 (MD5) Previous issue date: 2019-06-28","Embargo set by: Seth Robbins for item 112913 Lift date: 2021-11-26T20:49:41Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","U of I Only Restriction Lifted for Item 112913 on 2021-11-27T10:15:33Z."],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/105768"],"dc:language":["en"],"dc:rights":["Copyright 2019 Jianxiong Gao"],"dc:subject":["Symbolic Execution Computer Science Education Feedback Program Analysis"],"dc:title":["Use of symbolic execution as automated grading tool for introductory programming courses"],"dc:type":["text"],"thesis:degree_discipline":["Electrical & Computer Engr"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:45Z"}