Automated test case prioritization using machine learning for large scale continuous integration environments

Continuous Integration (CI) systems rely on automated testing to maintain the stability and quality of software. Test Case Prioritization (CI-TCP) is essential for improving the order of test execution and enabling faster failure detection. It is possible to enhance CITCP by improving the prediction of test failures and refining the prioritization process by leveraging machine learning (ML) models. However, an automated and scalable solution is required to address the complexity of feature extraction and selection, ML model tuning, and evaluation in large-scale CI environments. This thesis aims to integrate ML techniques to improve the prioritization process in large-scale CI systems. The core contribution is automating the selection of relevant features and tuning ML models with a continuous feedback loop for failure prediction and prioritization. By incorporating ML-based feature selection techniques, we ensure that the models are effective and adaptive to the varying configurations of CI environments. Furthermore, after implementing different ML models, our automation extends to hyperparameter tuning, where model performance is tuned without manual intervention and ensemble learning methods are employed to improve prediction capabilities further, showcasing the impact of our research. Overall, a comprehensive framework is required to automate the generation of MLbased prioritized lists in large-scale CI systems. Therefore, we present an end-to-end framework that thoroughly automates feature extraction and selection, hyperparameter tuning, ensemble learning, and the evaluation of ML models. We have designed this comprehensive framework to be practical and suitable for various prioritization problems that incorporate new ML solutions. Our framework has shown improvement in real-life large-scale CI systems, where we observed an increase of approximately 50% in the efficiency of prioritized test suites. Our experimental results also show significantly improved early failure detection and prioritization accuracy, outperforming traditional approaches and making it a valuable solution for large-scale CI environments.

Name thesis:degree_name

Doctor of Philosophy (PhD)

Discipline thesis:degree_discipline

Electrical and Computer Engineering

Grantor

University of Ontario Institute of Technology

Year dc:date.issued

2024

Author dc:creator

• Khan, Md Asif

Advisors dc:contributor.advisor

• Azim, Akramul
• Liscano, Ramiro

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/10155/1897

OAI identifier oai:identifier

oai:ontariotechu.scholaris.ca:10155/1897