{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/98348"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/98348","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Proactively detecting unreliable tests","abstract":"Regression testing is the most wide-spread method to ensure the quality of software systems. Whenever a change is made to the software, tests are run to ensure bugs are not introduced: if all tests pass, the change is merged into the codebase; otherwise, the developer needs to identify the bug that was introduced by the change. Developers assume that the outcomes of the tests in the regression testing process are reliable, i.e., that the failure indicates a bug introduced by the change. Unfortunately, unreliable tests manage to get into the test suite, slowing down the developers’ workflow and having developers debug not their software but rather the test code or infrastructure. This dissertation presents two techniques to enable developers to more easily and proactively detect and debug unreliable tests early, before they become a problem and slow down the development process. Developers write unreliable tests, which may pass on their machine but may at a later point fail because the environment changes. This dissertation presents a technique to detect when code makes wrong assumptions on underdetermined APIs. While these assumptions may hold in the current environment, they may not hold in the future, causing tests to fail. The technique, NonDex, detects such wrong assumptions by exploring different behaviors that may not manifest in the current implementation, but are allowed by the specification. Furthermore, the dissertation presents PolDet, a technique to detect when tests pollute their environment; such polluting tests can cause other tests to fail or pass seemingly nondeterministically because of the environment pollution rather than changes in the code. The two techniques enable developers to identify when they are introducing unreliability in their test suite and help identify the root causes of the unreliable tests. The results of the evaluation on open-source projects show the techniques are effective at identifying issues in open-source code and also that developers are eager to fix the issues and adopt the tools.","abstract_html":"Regression testing is the most wide-spread method to ensure the quality of software systems. Whenever a change is made to the software, tests are run to ensure bugs are not introduced: if all tests pass, the change is merged into the codebase; otherwise, the developer needs to identify the bug that was introduced by the change. Developers assume that the outcomes of the tests in the regression testing process are reliable, i.e., that the failure indicates a bug introduced by the change. Unfortunately, unreliable tests manage to get into the test suite, slowing down the developers’ workflow and having developers debug not their software but rather the test code or infrastructure. This dissertation presents two techniques to enable developers to more easily and proactively detect and debug unreliable tests early, before they become a problem and slow down the development process. Developers write unreliable tests, which may pass on their machine but may at a later point fail because the environment changes. This dissertation presents a technique to detect when code makes wrong assumptions on underdetermined APIs. While these assumptions may hold in the current environment, they may not hold in the future, causing tests to fail. The technique, NonDex, detects such wrong assumptions by exploring different behaviors that may not manifest in the current implementation, but are allowed by the specification. Furthermore, the dissertation presents PolDet, a technique to detect when tests pollute their environment; such polluting tests can cause other tests to fail or pass seemingly nondeterministically because of the environment pollution rather than changes in the code. The two techniques enable developers to identify when they are introducing unreliability in their test suite and help identify the root causes of the unreliable tests. The results of the evaluation on open-source projects show the techniques are effective at identifying issues in open-source code and also that developers are eager to fix the issues and adopt the tools.","abstract_has_math":false,"creators":["Gyori, Alex"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Marinov, Darko","Parthasarathy, Madhusudan","Xie, Tao","Lahiri, Shuvendu K"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2017,"date_issued":"2017-09-29T17:56:34Z","date_published":"2017-09-29T17:56:34Z","updated_at":"2026-07-22T22:24:35Z","subjects":["Unreliable tests","Flaky tests","Proactive detection"],"languages":["en"],"rights":["Copyright 2017 Alex Gyori"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/98348","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Marinov, Darko","Parthasarathy, Madhusudan","Xie, Tao","Lahiri, Shuvendu K"]},{"key":"dc:creator","label":"Author","values":["Gyori, Alex"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2017-09-29T17:56:34Z","2017-07-10","2017-08"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Unreliable tests","Flaky tests","Proactive detection"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2017 Alex Gyori"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/98348"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Regression testing is the most wide-spread method to ensure the quality of software systems. Whenever a change is made to the software, tests are run to ensure bugs are not introduced: if all tests pass, the change is merged into the codebase; otherwise, the developer needs to identify the bug that was introduced by the change. Developers assume that the outcomes of the tests in the regression testing process are reliable, i.e., that the failure indicates a bug introduced by the change. Unfortunately, unreliable tests manage to get into the test suite, slowing down the developers’ workflow and having developers debug not their software but rather the test code or infrastructure. This dissertation presents two techniques to enable developers to more easily and proactively detect and debug unreliable tests early, before they become a problem and slow down the development process. Developers write unreliable tests, which may pass on their machine but may at a later point fail because the environment changes. This dissertation presents a technique to detect when code makes wrong assumptions on underdetermined APIs. While these assumptions may hold in the current environment, they may not hold in the future, causing tests to fail. The technique, NonDex, detects such wrong assumptions by exploring different behaviors that may not manifest in the current implementation, but are allowed by the specification. Furthermore, the dissertation presents PolDet, a technique to detect when tests pollute their environment; such polluting tests can cause other tests to fail or pass seemingly nondeterministically because of the environment pollution rather than changes in the code. The two techniques enable developers to identify when they are introducing unreliability in their test suite and help identify the root causes of the unreliable tests. The results of the evaluation on open-source projects show the techniques are effective at identifying issues in open-source code and also that developers are eager to fix the issues and adopt the tools.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2017-09-29 without embargo terms","The student, Alex Gyori, accepted the attached license on 2017-07-10 at 10:20.","The student, Alex Gyori, submitted this Dissertation for approval on 2017-07-10 at 10:24.","This Dissertation was approved for publication on 2017-07-10 at 11:31.","DSpace SAF Submission Ingestion Package generated from Vireo submission #11347 on 2017-09-29 at 11:28:37","Made available in DSpace on 2017-09-29T17:56:34Z (GMT). No. of bitstreams: 2 GYORI-DISSERTATION-2017.pdf: 804215 bytes, checksum: 54ac6a5778090bb56708343c9fb95ecc (MD5) LICENSE.txt: 4207 bytes, checksum: 51c4670d254f0edd315f7d0a36063e85 (MD5) Previous issue date: 2017-07-10"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Proactively detecting unreliable tests"]}]}],"canonical_facts":{"dc:contributor":["Marinov, Darko","Parthasarathy, Madhusudan","Xie, Tao","Lahiri, Shuvendu K"],"dc:creator":["Gyori, Alex"],"dc:date":["2017-09-29T17:56:34Z","2017-07-10","2017-08"],"dc:description":["Regression testing is the most wide-spread method to ensure the quality of software systems. Whenever a change is made to the software, tests are run to ensure bugs are not introduced: if all tests pass, the change is merged into the codebase; otherwise, the developer needs to identify the bug that was introduced by the change. Developers assume that the outcomes of the tests in the regression testing process are reliable, i.e., that the failure indicates a bug introduced by the change. Unfortunately, unreliable tests manage to get into the test suite, slowing down the developers’ workflow and having developers debug not their software but rather the test code or infrastructure. This dissertation presents two techniques to enable developers to more easily and proactively detect and debug unreliable tests early, before they become a problem and slow down the development process. Developers write unreliable tests, which may pass on their machine but may at a later point fail because the environment changes. This dissertation presents a technique to detect when code makes wrong assumptions on underdetermined APIs. While these assumptions may hold in the current environment, they may not hold in the future, causing tests to fail. The technique, NonDex, detects such wrong assumptions by exploring different behaviors that may not manifest in the current implementation, but are allowed by the specification. Furthermore, the dissertation presents PolDet, a technique to detect when tests pollute their environment; such polluting tests can cause other tests to fail or pass seemingly nondeterministically because of the environment pollution rather than changes in the code. The two techniques enable developers to identify when they are introducing unreliability in their test suite and help identify the root causes of the unreliable tests. The results of the evaluation on open-source projects show the techniques are effective at identifying issues in open-source code and also that developers are eager to fix the issues and adopt the tools.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2017-09-29 without embargo terms","The student, Alex Gyori, accepted the attached license on 2017-07-10 at 10:20.","The student, Alex Gyori, submitted this Dissertation for approval on 2017-07-10 at 10:24.","This Dissertation was approved for publication on 2017-07-10 at 11:31.","DSpace SAF Submission Ingestion Package generated from Vireo submission #11347 on 2017-09-29 at 11:28:37","Made available in DSpace on 2017-09-29T17:56:34Z (GMT). No. of bitstreams: 2 GYORI-DISSERTATION-2017.pdf: 804215 bytes, checksum: 54ac6a5778090bb56708343c9fb95ecc (MD5) LICENSE.txt: 4207 bytes, checksum: 51c4670d254f0edd315f7d0a36063e85 (MD5) Previous issue date: 2017-07-10"],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/98348"],"dc:language":["en"],"dc:rights":["Copyright 2017 Alex Gyori"],"dc:subject":["Unreliable tests","Flaky tests","Proactive detection"],"dc:title":["Proactively detecting unreliable tests"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:35Z"}