Secure and Efficient In-Process Monitor and Multi-Variant Execution

Control flow hijacking attacks such as Return Oriented Programming (ROP) and data oriented attacks like Data Oriented Programming (DOP) are problems still plaguing modern software today. While there have been many attempts at hardening software and protecting against these attacks, the heavy performance cost of running these defenses and intrusive modifications required has proven to be a barrier to adoption. In this work, we present Monguard, a high-performance hardware assisted in-process monitor protection system utilizing Intel Memory Protection Keys (MPK) to enforce execute-only memory, combined with code randomization and runtime binary patching to effectively protect and hide in-process monitors. Next, we introduce L-MVX, a flexible lightweight Multi-Variant Execution (MVX) system running in the in-process monitor system that aims to solve some of the performance problems of recent MVX defenses through selective program call graph protection and in-process monitoring, maintaining security guarantees either by breaking attacker assumptions or creating a scenario where a particular attack only works on a single variant.

Name thesis:degree_name

Master of Science

Level thesis:degree_level

masters

Discipline thesis:degree_discipline

Computer Engineering

Department dc:contributor.department

Electrical and Computer Engineering

Grantor dc:publisher

Virginia Tech

Year dc:date.issued

2021

Author dc:creator

• Yeoh, SengMing

Chair dc:contributor.committeechair

• Ravindran, Binoy

Committee members dc:contributor.committeemember

• Wang, Haining
• Nikolaev, Ruslan

• Multi-Variant Execution
• In-Process Monitor
• Memory Isolation

Statement dc:rights

• In Copyright

Licence dc:rights.uri

• http://rightsstatements.org/vocab/InC/1.0/

Dc Identifier Other

vt_gsexam:28919

OAI identifier oai:identifier

oai:vtechworks.lib.vt.edu:10919/102158