Rethinking the Weakness of Stream Ciphers and Its Application to Encrypted Malware Detection

<p>Encryption key use is a critical component to the security of a stream cipher: because many implementations simply consist of a key scheduling algorithm and logical exclusive or (XOR), an attacker can completely break the cipher by XORing two ciphertexts encrypted under the same key, revealing the original plaintexts and the key itself. The research presented in this paper reinterprets this phenomenon, using repeated-key cryptanalysis for stream cipher identification. It has been found that a stream cipher executed under a fixed key generates patterns in each character of the ciphertexts it produces and that these patterns can be used to create a fingerprint which is distinct to a certain stream cipher and encryption key pair. A discrimination function, trained on this fingerprint, optimally separates ciphertexts generated through an enciphering pair from those which are generated by any other means. The patterns were observed in the Rivest Cipher 4 (RC4), ChaCha20-Poly1305, and Salsa20 stream ciphers as well as block cipher modes of operation that perform similarly to stream ciphers, such as: Counter (CTR), Galois/Counter (GCM), and Output feedback (OFB) modes. The discriminatory scheme proposed in this study perfectly detects ciphertexts of a fixed-key stream cipher with or without explicit knowledge of the key which may be utilized to detect a specific type of malware that exploits a stream cipher with a stored key to encrypt or obfuscate its activity. Finally, using real-world example of this type of malware, it is shown that the scheme is capable of detecting packets sent by the DarkComet remote access trojan, which utilizes RC4, with 100% accuracy in about 36 μs, providing a fast and highly accurate tool to aid in detecting malware using encryption.</p>

Name thesis:degree_name

Master of Science in Computer Science (MSCS)

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Year dc:date.available

2020

Authors dc:creator

• Stone, William T.
• Son, Junggab

Contributors dc:contributor

• Junggab Son
• Chih-Cheng Hung

• Malware detection
• encrypted malware
• machine learning
• RC4
• Darkcomet
• Computer Engineering

Repository record dc:identifier

https://digitalcommons.kennesaw.edu/cs_etd/52

OAI identifier oai:identifier

oai:digitalcommons.kennesaw.edu:cs_etd-1054