No Linux, No Problem: Fast and Correct Windows Binary Fuzzing via Target-embedded Snapshotting

Coverage-guided fuzzing remains today's most successful approach for exposing software security vulnerabilities. Speed is paramount in fuzzing, as maintaining a high test case throughput enables more expeditious exploration of programs—leading to faster vulnerability discovery. High-performance fuzzers exploit the Linux kernel's customizability to implement process snapshotting: fuzzing-oriented execution primitives that dramatically increase fuzzing throughput. Unfortunately, such speeds remain elusive on Windows. The closed-source nature of its kernel prevents current kernel-based snapshotting techniques from being ported—severely limiting fuzzing's effectiveness on Windows programs. Thus, accelerating vetting of the Windows software ecosystem demands a fast, correct, and kernel-agnostic fuzzing execution mechanism. We propose making state snapshotting an application-level concern as opposed to a kernel-level concern via target-embedded snapshotting. Target-embedded-snapshotting combines binary- and library-level hooking to allow applications to snapshot themselves—while leaving both their source code and the Windows kernel untouched. Our evaluation on 10 real-world Windows binaries shows that target-embedded snapshotting overcomes the speed, correctness, and compatibility challenges of previous Windows fuzzing execution mechanisms (i.e., process creation, forkserver-based cloning, and in-memory looping). The result is 7–182x increased performance.

Name thesis:degree_name

Master of Science

Level thesis:degree_level

masters

Discipline thesis:degree_discipline

Computer Science and Applications

Department dc:contributor.department

Computer Science and Applications

Grantor dc:publisher

Virginia Tech

Year dc:date.issued

2023

Author dc:creator

• Stone, Leo Calvin

Chair dc:contributor.committeechair

• Hicks, Matthew

Committee members dc:contributor.committeemember

• Meng, Na
• Hoang, Thang

• fuzzing
• security
• windows

Statement dc:rights

• In Copyright

Licence dc:rights.uri

• http://rightsstatements.org/vocab/InC/1.0/

Language dc:language.iso

en

Dc Identifier Other

vt_gsexam:36889

OAI identifier oai:identifier

oai:vtechworks.lib.vt.edu:10919/115124