Crafting unusual programs for fuzzing deep learning libraries

Deep Learning (DL) applications play a vital role in modern society. Bugs in DL libraries can significantly impact a wide range of downstream DL applications, making it crucial to develop effective testing techniques for these libraries. Generating valid input programs for fuzzing DL libraries is challenging, as they must adhere to both the syntax/semantics of the supported languages (e.g., Python) and the tensor/operator constraints required for constructing valid computational graphs. The recent TitanFuzz work has shown, for the first time, that modern Large Language Models (LLMs) can be directly employed to implic- itly learn all language and DL computation constraints to create valid programs for fuzzing DL libraries. However, LLMs tend to generate ordinary programs that follow patterns/to- kens similar to typical programs found in their vast training corpora (e.g., GitHub), whereas fuzzing favors unusual inputs that cover edge cases or are less likely to be manually produced. To address this challenge, we propose AtlasFuzz, the first technique to prime LLMs for synthesizing unusual programs to enhance fuzzing effectiveness. AtlasFuzz is based on the well-established hypothesis that historically bug-triggering programs may contain rare and valuable code elements crucial for bug discovery. While traditional techniques leveraging such historical information demand extensive human effort to design dedicated generators and ensure the syntactic/semantic validity of the generated programs, AtlasFuzz demon- strates that this process can be fully automated through the intrinsic capabilities of LLMs (including fine-tuning and in-context learning) and is generalizable and applicable to chal- lenging domains. Furthermore, AtlasFuzz also highlights the potential of directly utilizing the instruction-following capability of the recent ChatGPT for effective fuzzing. Our experimental study on two popular DL libraries (PyTorch and TensorFlow) reveals that AtlasFuzz is an effective fuzzer for DL libraries, detecting 18 bugs, including 10 already confirmed as previously unknown bugs.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Yang, Shujing

Contributors dc:contributor

• Zhang, Lingming

• Software Engineering
• Language Models

Statement dc:rights

• Copyright 2023 Shujing Yang

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/120389

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F120389