Testing and evaluation of machine learning’s capability of generating program code

With rapid advances in machine learning, large language models (LLMs) are increasingly used for code generation, yet systematically evaluating their capabilities remains challenging. This thesis proposes a datamorphic, scenario-based testing framework for LLM code generation, featuring an extensible and automatable system that integrates diverse analysis tools. Based on this framework, the thesis evaluates multiple quality attributes, including functional correctness, syntactic closeness, code complexity, and code smells, and further examines usability in multi-attempt code generation and robustness of LLMs. Overall, this thesis moves the field, from single-score evaluation to scenario-aware, multi-attribute, automated testing. Extensive experiments demonstrate the framework’s effectiveness and reveal previously unreported behaviors of state-of-the-art LLMs. To support scenario-based evaluation, this work introduces ScenEval, a large-scale benchmark of 12,864 Java programming tasks with reference solutions and metadata. A bidirectional JUnit-based testing framework is proposed to detect both omission and commission errors, improving the reliability of functional correctness evaluation. Motivated by Ziegler et al.’s observation that developers value a code’s potential for further development, this thesis investigates syntactic closeness with respect to user intent. Results show that structural similarity metrics based on ASTs, DFGs, and PDGs strongly correlate with functional correctness, making them effective measures of syntactic closeness. The thesis also presents a systematic analysis of code complexity and code smells in LLM-generated programs. Using size-based, cyclomatic, and cognitive metrics, it finds that correctly generated code is often more complex than professional reference solutions, while incorrect code from some models tends to be simpler. Moreover, LLM-generated programs exhibit more code smell violations, which increase with task complexity. For robustness evaluation, the thesis introduces scenario domain analysis, a micro-level method showing that state-of-the-art LLMs remain robust under mild paraphrasing of task descriptions. Finally, the usability in multi-attempt generation process is automated using functional correctness and code smell feedback for prompt refinement and termination. Results indicate that while LLMs are usable for simpler tasks, usability degrades significantly for advanced and high-complexity programming tasks.

Grantor dc:publisher

Oxford Brookes University

Author dc:creator

• Ghosh Paul, Debalina

Contributors dc:contributor

• Zhu, Hong
• Bayley, Ian

Statement dc:rights

• All rights reserved

Language dc:language

en

DOI dc:identifier

https://doi.org/10.24384/h46a-zg76

OAI identifier oai:identifier

tle:57c10d80-1ac4-4d30-bc58-a1a320c8bebc:d6bd9758-527a-46cd-bfe2-c433766e8fca:1