Exploring large language models as configuration validators: Techniques, challenges, and opportunities

Misconfigurations are major causes of software failures. Existing practices rely on developer-written rules or test cases to validate configurations, which are expensive to implement and maintain, and are hard to be comprehensive. Machine learning (ML) for configuration validation is considered a promising direction, but has been facing challenges such as the need of large-scale field data and system-specific models, which are hard to generalize. Recent advances in Large Language Models (LLMs) show promise in addressing some of the long-lasting limitations of ML-based configuration validation. This thesis presents a first analysis on the feasibility and effectiveness of using LLMs for configuration validation. We empirically evaluate LLMs as configuration validators by developing a generic LLM-based configuration validation framework, named Ciri. Ciri employs effective prompt engineering with few-shot learning based on both valid configuration and misconfiguration data. Ciri checks outputs from LLMs when producing results, addressing hallucination and nondeterminism of LLMs. We evaluate Ciri’s validation effectiveness on eight popular LLMs using configuration data of ten widely deployed open-source systems. Our analysis (1) confirms the potential of using LLMs for configuration validation, e.g., Ciri with Claude-3-Opus detects 45 out of 51 real-world misconfigurations, outperforming recent configuration validation techniques. (2) explores design space of LLM-based validators like Ciri, especially in terms of prompt engineering with few-shot learning and voting. We find that using configuration data as shots can enhance validation effectiveness. (3) reveals open challenges: Ciri struggles with certain types of misconfigurations such as dependency violations and version-specific misconfigurations. It is also biased to the popularity of configuration parameters, causing both false positives and false negatives. We discuss the promising directions to address these challenges and further improve Ciri. Chain-of-Thoughts (CoT) can mimic the reasoning process of a human expert, which makes the validation more transparent and potentially more accurate. Additionally, LLMs can generate environment-specific scripts to run in the target environment, that can help identify issues like misconfigured paths, unreachable addresses, missing packages, and invalid permissions. We also plan to explore extending Ciri into a multi-agent framework, where Ciri can interact with additional tools such as Ctest and Cdep through agent frameworks.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Lian, Xinyu

Contributors dc:contributor

• Xu, Tianyin
• Marinov, Darko

• Software Engineering
• Machine Learning

Statement dc:rights

• Copyright 2024 Xinyu Lian

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124552

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124552