Comparative research on code vulnerability detection: Open-source vs. proprietary large language models and LSTM neural network

The reliance of industries such as banking, e-commerce, logistics, transportation, energy, and healthcare on computer systems has escalated the threat of cyberattacks. With the rapid development of the online ecosystem and the growth of sensitive data, cybercriminals have developed the ability to attack software systems. Safeguarding software has become an urgent yet challenging task despite advancements in technology. The contributions of this thesis show the incorporation of security into the Software Development Lifecycle through a focus on Static Code Analysis as a proactive approach for the discovery and mitigation of security flaws in the development cycle. This research focuses on enhancing vulnerability detection in source code through advanced machine learning techniques, including open-source fine-tuned and proprietary large language models. It compares the models, including CodeGen2, LLaMA 2, and GPT, developed by OpenAI to quantitative and qualitative how suitable they are for detecting vulnerabilities, their accuracy and efficiency. Using a zero-shot classification-based approach, the study examines these models’ capabilities in detecting security risks and compares them to Word2Vec+LSTM neural networks. The findings reveal that CodeGen2 emerges as the most reliable model for vulnerability detection, achieving near-perfect precision and balanced recall, leading to superior F1 and AUC scores. LLaMA2-7b delivers reasonable performance, particularly in precision, but falls short in recall. Conversely, GPT-4 Assistant excels in recall but suffers from a high false positive rate, limiting its effectiveness. Classical neural networks, such as Word2Vec+LSTM, demonstrate moderate capability but lag modern LLMs in precision and recall. Through this comparative analysis, the thesis underscores the importance of selecting tools aligned with organizational needs and constraints. The insights gained contribute to developing secure software by integrating machine learning into the SDLC, inspiring further progress in vulnerability detection and mitigating risks in an increasingly digital world.

Name thesis:degree_name

Master of Applied Technologies (Computing)

Level thesis:degree_level

Masters

Grantor

Unitec, Te Pūkenga – New Zealand Institute of Skills and Technology

Year dc:date.issued

2024

Author dc:creator

• Don, Ravihansa Geekiyanage Geekiyanage

• software development
• vulnerability assessment
• risk management framework
• software security
• computer security
• Long Short-Term Memory (LSTM)
• open source
• neural networks
• large language models

Statement dc:rights

• All rights reserved

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/10652/6749

OAI identifier oai:identifier

oai:researchbank.ac.nz:10652/6749