Leveraging Retrieval-Augmented Generation, Prompt Engineering, and Vision Language Models for Surface Defect Classification and Root Cause Analysis in Manufacturing

This study investigates the integration of Vision Language Model (VLM)s, Retrieval-Augmented Generation (RAG), and structured prompt-engineering strategies to enhance surface defect classification and Root Cause Analysis (RCA) in manufacturing. The research examines whether generative and multimodal Artificial Intelligence (AI) systems can deliver accurate, explainable, and cost efficient quality control for manufactured components. The first study assessed GPT-4o’s capability for surface defect detection and classification using two datasets: the MVTec AD screw subset and a custom #3DBenchy dataset developed by the author. Results showed that image-only prompting consistently outperformed multimodal (image + text) inputs, achieving up to 94.4% defect identification and 82.6% classification accuracy on the #3DBenchy dataset, and 90.5% defect identification with 76% classification accuracy on the MVTec AD screw subset. Small-sample fine-tuning (5–30 images per class) further improved classification accuracy across both datasets. The second study extended the framework to RCA using RAG and structured prompting, incorporating domain-specific manufacturing knowledge to improve diagnostic reasoning. Three GPT-5 model variants (gpt-5-nano, gpt-5-mini, and gpt-5) were evaluated across reasoning effort levels and prompting strategies (ReAct and ReAct + 5 Whys). Larger models achieved higher diagnostic accuracy, whereas smaller variants—particularly gpt-5-nano at low reasoning effort—offered optimal cost-efficiency. Inconsistent RAG performance underscored the need for curated retrieval contexts. The final study introduced a gated ReAct-style RCA pipeline applying sequential reasoning across the Man, Method, and Machine categories of the 6M framework. Incorporating multimodal data such as G-code parameters, operator logs, temperature time-series, and bed-mesh images enabled near-perfect parameter-level diagnosis. Token usage scaled predictably with reasoning depth, validating the trade-off between interpretability and computational efficiency. Overall, the findings confirm that VLMs, when combined with structured prompting and domain-specific retrieval, can serve as reliable tools for automated defect diagnosis and causal reasoning—laying the groundwork for scalable, explainable, and data-driven quality control systems in advanced manufacturing.

Grantor dc:publisher

Stellenbosch : Stellenbosch University

Year dc:date.issued

2026

Author dc:creator

• Mocke, Johannes Jacobus

Advisors dc:contributor.advisor

• Louw, Louis
• Lucke, D.

Language dc:language.iso

en

Repository record dc:identifier.uri

https://scholar.sun.ac.za/handle/10019.1/136200

OAI identifier oai:identifier

oai:scholar.sun.ac.za:10019.1/136200