Monitoring security of enterprise hosts via DNS data analysis

Enterprise Networks are growing in scale and complexity, with heterogeneous connected assets needing to be secured in different ways. Nevertheless, virtually all connected assets use the Domain Name System (DNS) for address resolution. Thus DNS has become a convenient vehicle for attackers to covertly perform Command and Control (C&C) communication, data theft, and service disruption across a wide range of assets. Enterprise security appliances that monitor network traffic typically allow all DNS traffic through as it is vital for accessing any web service; they may at best match against a database of known malicious patterns, and are therefore ineffective against zero-day attacks. This thesis focuses on three high-impact cyber-attacks that leverage DNS, specifically data exfiltration, malware C&C communication, and service disruption. Using big data (over 10B packets) of DNS network traffic collected from a University campus and a Government research organization over six months, we illustrate the anatomy of these attacks, train machines for automatically detecting such attacks, and evaluate their efficacy in the field. The contributions of this thesis are three-fold: Our first contribution tackles data exfiltration using DNS. We analyze outgoing DNS queries to identify many stateless attributes such as the number of characters, the number of labels, and the entropy of the domain name to distinguish malicious data exfiltration queries from legitimate ones. We train our machines using ground-truth obtained from a public list of top 10K legitimate domains and empirically validate and tune our models to achieve over 98% accuracy in correctly distinguish legitimate DNS queries from malicious ones, the latter coming from known malware domains as well as synthetically generated using popular DNS exfiltration tools. Our second contribution tackles malware C&C communication using DNS. We analyze DNS outgoing queries to identify more than twenty families of DGA (Domain Generation Algorithm)-enabled malware when communicating with their C&C servers. We identify attributes of network traffic that commences following the resolution of a DGA-based DNS query. We train three protocol-specific one-class classifier models, for HTTP, HTTPS and UDP flows, using public packet traces of known malware. We develop a monitoring system that uses reactive rules to automatically and selectively mirror TCP/UDP flows (between internal hosts and malware servers) pertinent to DGA queries for diagnosis by the trained models. We deploy our system in the field and evaluate its performance to show that it flags more than 2000 internal assets as potentially infected, generating more than a million suspicious flows, of which more than 97% are verified to be malicious by an off-the-shelf intrusion detection system. Our third contribution studies the use of DNS for service disruption. We analyze incoming DNS messages, with a specific focus on non-existent (NXD) DNS responses, to distinguish benign from malicious NXDs. We highlight two attack scenarios based on their requested domain names. Using NXD behavioral attributes of internal hosts, we develop multi-staged iForest classification models to detect internal hosts launching service disruption attacks. We show how our models can detect infected hosts that generate high-volume and low-volume distributed NXD-based attacks on public resolvers and/or authoritative name servers with an accuracy of over 99% in correctly classifying legitimate hosts. Our work shines a light on a critical vector in enterprise security and equips the enterprise network operator with the means to detect and block sophisticated attackers who use DNS as a vehicle for malware C&C communication, data exfiltration, and service disruption.

Grantor dc:publisher

UNSW, Sydney

Year dc:date

2021

Author dc:creator

• Ahmed, Jawad

• DNS Exfiltration
• DNS Security
• Programmable Networks
• Domain Generation Algorithms

Statement dc:rights

• open access
• CC BY-NC-ND 3.0
• free_to_read

Licence

• https://purl.org/coar/access_right/c_abf2
• https://creativecommons.org/licenses/by-nc-nd/3.0/au/

Language dc:language

EN

Identifier

https://doi.org/10.26190/unsworks/22739

OAI identifier oai:identifier

oai:unsworks.library.unsw.edu.au:1959.4/71128