{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/129317"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/129317","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"A novel approach to repair non-idempotent-outcome tests with an LLM-based agent","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-10-19 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-10-19 without embargo terms","abstract_has_math":false,"creators":["Ke, Kaiyao"],"institution":"University of Illinois Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Marinov, Darko"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025-05-05","date_published":"2025-05-05","updated_at":"2026-07-22T22:25:04Z","subjects":["Software Testing","Flaky Tests","Software Quality Assurance"],"languages":["en","eng"],"rights":["Copyright 2025 Kaiyao Ke"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/129317","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Marinov, Darko"]},{"key":"dc:creator","label":"Author","values":["Ke, Kaiyao"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2025-05-05","2025-05"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Software Testing","Flaky Tests","Software Quality Assurance"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2025 Kaiyao Ke"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/129317"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-10-19 without embargo terms","The student, Kaiyao Ke, accepted the attached license on 2025-05-04 at 14:11.","The student, Kaiyao Ke, submitted this Thesis for approval on 2025-05-04 at 14:18.","This Thesis was approved for publication on 2025-05-05 at 12:55.","DSpace SAF Submission Ingestion Package generated from Vireo submission #22201 on 2025-10-19 at 18:12:30","Flaky tests, characterized by inconsistent results across repeated executions, present significant challenges in software testing, especially during regression testing. Recently, there has been emerging research interest in non-idempotent-outcome (NIO) flaky tests—tests that pass on the initial run but fail on subsequent executions within the same environment. Despite progress in utilizing Large Language Models (LLMs) to address flaky tests, existing methods have not tackled NIO flaky tests. The limited context window of LLMs restricts their ability to incorporate relevant source code beyond the test method itself, often overlooking crucial information needed to address state pollution, which is the root cause of NIO flakiness. This thesis introduces NIODebugger, the first framework to utilize an LLM-based agent to repair flaky tests. NIODebugger features a three-phase design: detection, exploration, and fixing. In the detection phase, dynamic analysis collects stack traces and custom test execution logs from multiple test runs, which helps in understanding accumulative state pollution (for example, a counter is 2 during the first rerun and becomes 3 in the second rerun). During the exploration phase, the LLM-based agent provides instructions for extracting relevant source code associated with test flakiness. In the fixing phase, NIODebugger repairs the tests using the information gathered from the previous phases. NIODebugger can be integrated with multiple LLMs, achieving patching success rates ranging from 11.63% to 58.72%. Its best-performing variant, NIODebugger-GPT-4, successfully generated correct patches for 101 out of 172 previously unknown NIO tests across 20 large-scale open-source projects. We submitted pull requests for all generated patches; 58 have been merged, only 1 was rejected, and the remaining 42 are pending. The Java implementation of NIODebugger is provided as a Maven plugin accessible at https://github.com/kaiyaok2/NIOInspector. This thesis is based on the paper “NIODebugger: A Novel Approach to Repair Non-Idempotent-Outcome Tests with LLM-Based Agent” accepted to ICSE ’25."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["A novel approach to repair non-idempotent-outcome tests with an LLM-based agent"]}]}],"canonical_facts":{"dc:contributor":["Marinov, Darko"],"dc:creator":["Ke, Kaiyao"],"dc:date":["2025-05-05","2025-05"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-10-19 without embargo terms","The student, Kaiyao Ke, accepted the attached license on 2025-05-04 at 14:11.","The student, Kaiyao Ke, submitted this Thesis for approval on 2025-05-04 at 14:18.","This Thesis was approved for publication on 2025-05-05 at 12:55.","DSpace SAF Submission Ingestion Package generated from Vireo submission #22201 on 2025-10-19 at 18:12:30","Flaky tests, characterized by inconsistent results across repeated executions, present significant challenges in software testing, especially during regression testing. Recently, there has been emerging research interest in non-idempotent-outcome (NIO) flaky tests—tests that pass on the initial run but fail on subsequent executions within the same environment. Despite progress in utilizing Large Language Models (LLMs) to address flaky tests, existing methods have not tackled NIO flaky tests. The limited context window of LLMs restricts their ability to incorporate relevant source code beyond the test method itself, often overlooking crucial information needed to address state pollution, which is the root cause of NIO flakiness. This thesis introduces NIODebugger, the first framework to utilize an LLM-based agent to repair flaky tests. NIODebugger features a three-phase design: detection, exploration, and fixing. In the detection phase, dynamic analysis collects stack traces and custom test execution logs from multiple test runs, which helps in understanding accumulative state pollution (for example, a counter is 2 during the first rerun and becomes 3 in the second rerun). During the exploration phase, the LLM-based agent provides instructions for extracting relevant source code associated with test flakiness. In the fixing phase, NIODebugger repairs the tests using the information gathered from the previous phases. NIODebugger can be integrated with multiple LLMs, achieving patching success rates ranging from 11.63% to 58.72%. Its best-performing variant, NIODebugger-GPT-4, successfully generated correct patches for 101 out of 172 previously unknown NIO tests across 20 large-scale open-source projects. We submitted pull requests for all generated patches; 58 have been merged, only 1 was rejected, and the remaining 42 are pending. The Java implementation of NIODebugger is provided as a Maven plugin accessible at https://github.com/kaiyaok2/NIOInspector. This thesis is based on the paper “NIODebugger: A Novel Approach to Repair Non-Idempotent-Outcome Tests with LLM-Based Agent” accepted to ICSE ’25."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/129317"],"dc:language":["en","eng"],"dc:rights":["Copyright 2025 Kaiyao Ke"],"dc:subject":["Software Testing","Flaky Tests","Software Quality Assurance"],"dc:title":["A novel approach to repair non-idempotent-outcome tests with an LLM-based agent"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:04Z"}