REFORM: A framework for malware packer analysis using information theory and statistical methods

Sun, L 2010, REFORM: A framework for malware packer analysis using information theory and statistical methods, Doctor of Philosophy (PhD), Mathematical and Geospatial Sciences, RMIT University.

Document type: Thesis
Collection: Theses

Attached Files
Name Description MIMEType Size
Sun.pdf Thesis application/pdf 16.94MB
Title REFORM: A framework for malware packer analysis using information theory and statistical methods
Author(s) Sun, L
Year 2010
Abstract Malware (malicious software) is a term used to describe computer viruses, Trojan horses, and other pieces of software that are used to attack computer systems. The increasing outbreak of malware in recent years poses a serious security threat to computer networks.

Malware writers often obfuscate malware to hinder malware scanners from malicious code detection, i.e., to hide the fact that the software is actually malicious. Packing is the most common obfuscation method used by malware writers. Recently, there has been a dramatic increase in the number of new packers and variants of existing ones. Moreover, packers are employing increasingly sophisticated anti-unpacker tricks and obfuscation methods.

Identifying a packer and obtaining a sample of unpacked malware are important to AV (Anti-virus) researchers who work on updating antivirus software to defend against malware, so that they can perform in-depth analysis. However, packer analysis is a technically intense research task, requiring the AV experts' deep knowledge of hardware, operating systems, compilers and programming languages. The significant growth of packers, in both number and complexity, prevents AV researchers from carrying out their daily AV research work efficiently and effectively.

This PhD project has investigated the common features of packers and presented a novel, fast yet effective packer analysis framework called REFORM (Reverse Engineering For Obfuscation ReMoval). The system applies various technologies including reverse engineering, compression algorithms and statistical methods to de-obfuscate packers.

REFORM is comprised of three major components that solve the problem of automatic packer analysis at three important stages of the packer analysis life cycle, namely packer detection, packer identification and unpacking, respectively: (1) It incorporates a novel randomness test that preserves local detail in the packer. This makes it easy for an AV researcher to distinguish areas of compressed/encrypted data from other code and data. (2) Using the above randomness test, each packer is seen to exhibit a unique pattern in its randomness distribution. The REFORM framework therefore provides an extremely effective packer classification model based on a set of randomness measurements generated from a packed file. Various statistical classifiers have also been integrated in REFORM to achieve even better classification performance. (3) REFORM enables an efficient generic unpacking strategy which uses an ordered address execution histogram to capture the memory after the unpacking loop has executed.

We demonstrate REFORM 's capability on speeding up packer detection, identification and unpacking procedures. Such an automatic system is shown in the thesis to be essential to keeping up with the accelerating growth in packed malware.
Degree Doctor of Philosophy (PhD)
Institution RMIT University
School, Department or Centre Mathematical and Geospatial Sciences
Keyword(s) malware
packer analysis
packer classification
pattern recognition
information theory
statistical method
Version Filter Type
Access Statistics: 754 Abstract Views, 2887 File Downloads  -  Detailed Statistics
Created: Tue, 16 Nov 2010, 17:12:03 EST
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us