Evaluation, Interpretation, and Maintenance of Machine Learning Models for IT Operations

AIOps (Artificial Intelligence for IT Operations) solutions leverage the massive data generated during the operation of large-scale systems and machine learning models to assist in managing system operations. While prior studies focus on innovative modeling techniques to improve the performance of AIOps models, how to smoothly transition AIOps solutions from development to production remains an underexplored topic. Since operational data instances often exhibit temporal dependencies, the use of improper model evaluation methods can lead to performance overestimation. Insufficient model maintenance on AIOps solutions incorporated in the production environment can also lead to future performance degradation. In addition, the consistency of model interpretation is impacted by the volatile nature of operational data. These threats pose significant challenges for lab-developed AIOps solutions when deployed in production environments. Therefore, this thesis proposes to explore related techniques for mitigating the challenges and helping practitioners make better decisions for deploying AIOps solutions in dynamic operational environments. We evaluate the impact of different data splitting decisions to understand the data leakage and concept drift challenges in the model evaluation stage. Our findings motivate practitioners to take precautions against using the random data splitting method that could induce data leakage. We assess the factors that impact the consistency of AIOps model interpretations. We propose guidelines for practitioners that help derive more reliable and consistent interpretations from AIOps models. We evaluate model update strategies for maintaining AIOps solutions in terms of performance, updating cost, and stability. Our findings suggest that practitioners consider more sophisticated model update strategies to mitigate the impact of concept drift and to minimize operational cost and performance variation. We examine model selection mechanisms on historical models in maintaining AIOps solutions. Our findings highlight a potential research opportunities for model maintenance beyond the simple retraining and replacing strategies. This thesis helps practitioners manage and mitigate the above-mentioned challenges in the evaluation, interpretation, and maintenance of AIOps solutions to ensure the smooth transition from development to production.

Department dc:contributor.department

Computing

Year dc:date.issued

2025

Author dc:creator

• Lyu, Yingzhe

Advisor dc:contributor.supervisor

• Hassan, Ahmed E.

• Software Engineering
• AIOps
• Model Maintenance
• Failure Prediction
• Model Interpretation

Statement dc:rights

• Attribution-NonCommercial-ShareAlike 4.0 International

Licence dc:rights.uri

• http://creativecommons.org/licenses/by-nc-sa/4.0/

Language dc:language.iso

eng

Handle dc:identifier.uri

https://hdl.handle.net/1974/35984

OAI identifier oai:identifier

oai:queensu.scholaris.ca:1974/35984