Beyond rules: how Large Language Models are redefining cryptographic misuse detection

The use of Large Language Models (LLMs) in software development is rapidly growing, with developers increasingly relying on these models for coding assistance, including security-critical tasks. Our work presents a comprehensive comparison between traditional static analysis tools for cryptographic API misuse detection—CryptoGuard, CogniCrypt, and Snyk Code—and the LLMs—GPT, Llama, Claude, and Gemini. Using benchmark datasets (OWASP, CryptoAPI, and MASC), we evaluate the effectiveness of each tool in identifying cryptographic misuses. Our findings show that GPT 4-o-mini surpasses current state-of-the-art static analysis tools on the CryptoAPI and MASC datasets, though it lags on the OWASP dataset. Additionally, we assess the quality of LLM responses to determine which models provide actionable and accurate advice, giving developers insights into their practical utility for secure coding. This study highlights the comparative strengths and limitations of static analysis versus LLM-driven approaches, offering valuable insights into the evolving role of AI in advancing software security practices.

Name thesis:degree_name

Master of Science (MSc)

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Ontario Institute of Technology

Year dc:date.issued

2025

Author dc:creator

• Masood, Zohaib

Advisor dc:contributor.advisor

• Vargas Martin, Miguel

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/10155/1934

OAI identifier oai:identifier

oai:ontariotechu.scholaris.ca:10155/1934