Design and development of a vulnerability detection framework using artificial intelligence for embedded systems

Embedded systems play a critical role in industrial control, IoT, autonomous systems, and critical infrastructure. However, their widespread adoption introduces security vulnerabilities across three layers: the application layer, the embedded OS layer, and the hardware layer. This thesis presents a comprehensive approach to vulnerability detection and mitigation across these layers, leveraging machine learning, deep learning, and AI-driven solutions. At the application layer, this research focuses on low-level programming applications that control or impact embedded systems, such as firmware, system utilities, and real-time control software. We develop deep learning-based models for automated vulnerability detection, evaluating Convolutional Neural Networks (CNNs) and BERT-based architectures. For the embedded OS layer, we introduced the EVDD dataset (Embedded Vulnerability Detection) to enhance Linux kernel vulnerability detection. Using big data processing, we construct a balanced dataset to improve machine learning models for intrusion detection and OS-level security analysis. Deep neural networks (DNNs) are implemented to detect and mitigate attacks, supporting effective threat detection under resource-constrained environments. At the hardware layer, we investigate vulnerabilities in FPGA based systems, such as bitstream manipulation and HDL coding flaws. We introduce the BitVul-LLM model for detecting FPGA bitstream vulnerabilities, and SecureLLAMA, a transformer-based framework using the FPGAvul dataset to identify hardware security risks. These models automate threat detection and significantly enhance hardware-level security. To improve detection across all layers, we developed an integrated framework using LLama 3, combining our models into a unified AI-driven system. This framework classifies vulnerabilities as hardware, OS-level, or application-level threats, improving precision and response strategies. By correlating vulnerabilities across layers, it enables advanced security assessment, efficient threat mitigation, and proactive defense mechanisms.

Name thesis:degree_name

Doctor of Philosophy (PhD)

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Ontario Institute of Technology

Year dc:date.issued

2025

Author dc:creator

• Alqarni, Mansour

Advisor dc:contributor.advisor

• Azim, Akramul

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/10155/1981

OAI identifier oai:identifier

oai:ontariotechu.scholaris.ca:10155/1981