Anomaly based Intrusion Detection System through Remote Virtual Machine Introspection

<p>Research on identifying malicious applications is an important direction in information security, especially when it comes to detection of evasive malware such as keyloggers, trojans, rootkits and their derivatives. Inspired by a biological immune system and based on negative selection algorithm approach to detect various types of malwares is proposed in this paper.</p> <p>By deeply studying Linux kernel, understanding links behind different internal system processes, examining, and experimenting with hundreds of various keyloggers we propose a single Artificial Intelligence based solution as a comprehensive protection against wide range of malwares. Developed Intrusion Detection System (IDS) can be deployed in the host operating system or launched remotely to introspect Virtual Machines (VMs) protecting them against many types of malicious software such as keyloggers, spyware/adware, rootkits, worms, trojans and other villainous threats. Additional research has been conducted to demonstrate application of proposed IDS in Multi-access edge computing (MEC) environment. As part of the research, there have been developed a custom keylogger with the ability to remotely control its behavior. This replication and continuous tests enabled discovery of 26 systemwide features such as system interrupts, triggered system calls, TCP and UDP signals, socket information, keyboard drivers’ characteristics, size of resident file mappings, and other parameters that are activated based on malware’s actions. Experiments with partially and fully activated features have been continuously conducted specifically against following malicious applications: Logkeys, Rootkit, Stitch, Blueberry, TrojanX, Jynx2, Jynxkit, Umbreon-Rootkit, Vlany.</p> <p>Developed lightweight and secure Virtual Machine Introspection (VMI) module – <em>KVMonitor</em> has been customized to return data in a JSON-like structure, which in return can be processed by the IDS. Proposed IDS implements Negative Selection Algorithm that can learn to detect multiple variations of malware based on data received from the VMI module. Deployed DEAP Python library as a replacement to custom built Evolutionary Algorithm facilitated generation of random detectors and improved detection accuracy by 30%. Initially developed to detect only keyloggers, IDS now is a cross-platform application with user-friendly graphical interface and ability to successfully detect anomalies triggered by wide range of malware.</p> <p>To organize real-world experiments, there has been developed a cross-continental testbed environment connecting networks between two labs at Kyushu Institute of Technology in Japan and The City College of New York through secure GRE tunnel. Conducted experiments on this testbed against hundreds of evasive malicious applications demonstrated average anomaly detection rate close to 97%.</p>

Name thesis:degree_name

Doctor of Philosophy (Ph.D.)

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Engineering

Year dc:date.available

2023

Author dc:creator

• Huseynov, Huseyn

Contributors dc:contributor

• Tarek Saadawi

• Intrusion Detection
• security and privacy
• virtualization
• artificial immune system
• cloud security
• Computational Engineering
• Computer and Systems Architecture
• Digital Communications and Networking
• Information Security
• OS and Networks

Repository record dc:identifier

https://academicworks.cuny.edu/cc_etds_theses/1149

OAI identifier oai:identifier

oai:academicworks.cuny.edu:cc_etds_theses-2177