Regression test prioritization for modern software

Continuous Integration (CI) is a common practice in software development for increasing code quality. CI runs regression test suite on each code change to help software developers find faults in the change. As codebase size and code change frequency have grown rapidly in modern software development, regression test suite runtime has also increased, thereby preventing timely debugging feedback to developers. Regression Test Prioritization (RTP) aims to expose faults in code change sooner by reordering tests in the test suite, so that the ones likely to fail are run earlier. Despite that RTP has been extensively studied for nearly three decades, research results of RTP are rarely put into practice. To facilitate the adoption of RTP for modern software systems, RTP must be demonstrated effective in speeding up regression fault detection in relevant and practical testing settings. Moreover, robust and efficient RTP tooling must be available for practitioners to easily apply RTP research results in popular software ecosystems. This dissertation takes a step forward in addressing these challenges via three lines of work. First, this dissertation examines key research findings from prior RTP studies on a newly-proposed dataset of long-running test suites with real test failures and up-to-date CI practices. Through extensive evaluation, this dissertation identifies the most effective and efficient RTP techniques under the impact of realistic CI issues such as flaky test failures. Second, this dissertation applies traditional RTP techniques for configuration testing, an emerging but critical testing scenario; it further proposes novel and effective RTP techniques for configuration testing. This dissertation assesses various factors that influence the effectiveness of RTP in speeding up misconfiguration detection, and provides detailed analyses and guidelines in applying RTP to configuration tests. Third, this dissertation presents pytest-ranking, a readily-usable open-source RTP tool for Python and its most popular testing framework Pytest. This dissertation realistically evaluates the efficacy of pytest-ranking, demonstrating that pytest-ranking is well integrated with Python/Pytest ecosystems, can be conveniently deployed into CI, has low overhead, and can detect faults substantially faster than baselines.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Cheng, Runxiang

Contributors dc:contributor

• Marinov, Darko
• Zhang, Lingming
• Xu, Tianyin
• Yang, Wei

• Software engineering
• Software testing
• Regression testing
• Test prioritization

Statement dc:rights

• Copyright 2025 Runxiang Cheng

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129204

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129204