Detecting, characterizing, and taming flaky tests

As software evolves, developers typically perform regression testing to ensure that their code changes do not break existing functionalities. During regression testing, developers can waste time debugging their code changes because of spurious failures from flaky tests, which are tests that non-deterministically pass or fail on the same code. These spurious failures mislead developers about their code changes because the failures are often due to bugs that existed before the code changes. One prominent category of flaky tests is order-dependent (OD) flaky tests. Each OD test has at least one order in which the test passes and another order in which the test fails, and for every test order, the test either passes or fails in all runs of that test order. Another prominent category is async-wait (AW) flaky tests. Each AW test makes at least one asynchronous call and passes if the asynchronous call finishes on time but fails if the call finishes too early or too late. This dissertation tackles three main aspects of flaky tests. First, this dissertation presents novel techniques to detect flaky tests so that developers can preemptively prevent the problem of flaky tests from affecting their regression testing results. Second, this dissertation presents novel techniques to characterize flaky tests to help developers better understand their flaky tests and to help researchers invent new solutions to the flaky-test problem. Lastly, this dissertation presents novel techniques to tame the problem of flaky tests by accommodating the flakiness so that flaky tests do not mislead developers during regression testing. For detecting flaky tests, this dissertation presents (1) iDFlakies, a framework for detecting and partially classifying flaky tests, and IDoFT, an increasingly used dataset of flaky tests found in popular open-source projects; (2) an analysis of the probability to detect OD tests from randomizing test orders and a novel algorithm to systematically explore all consecutive pairs of tests, guaranteeing to detect all OD tests that depend on one other test; and (3) a large-scale longitudinal study of flaky tests to determine when flaky tests become flaky and what changes cause them to become flaky—the results provide guidelines for when and how developers should spend their efforts to detect flaky tests. For characterizing flaky tests, this dissertation presents (1) the first automated technique to help developers debug flaky-test failures; and (2) a study to understand the effect that test orders have on non-deterministic tests, which can pass and fail even for the same test order—the results suggest that many of these tests can fail with significantly different failure rates for different test orders. Lastly, for taming flaky tests, this dissertation presents (1) the first automated techniques to reduce the number of spurious failures from OD tests, reducing such failures by 73%; and (2) the first automated techniques to speed up AW flaky tests while also keeping the number of spurious failures low, speeding up such tests by 38%. Overall, the work in this dissertation has helped detect more than 2000 flaky tests in over 150 open-source projects and fix more than 500 flaky tests in over 80 open-source projects.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2022

Author dc:creator

• Lam, Wing

Contributors dc:contributor

• Xie, Tao
• Marinov, Darko
• Nath, Suman
• Xu, Tianyin

• Software engineering
• software testing
• regression testing
• flaky tests

Statement dc:rights

• Copyright 2021 Wing Lam

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/113017

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/113017