| Abstract [eng] |
Machine learning-based malware detectors have significantly improved the ability to identify malicious software. However, the attackers have responded by using machine learning to obfuscate malware and evade detection. As such, for each new detection model that is deployed, more advanced adversarial malware generators eventually emerge to defeat it. This continuous cat-and-mouse game requires researchers to study adversarial obfuscation methods. Gaining a better understanding of how different methods improve evasion probability allows researchers to strengthen malware defenses and anticipate the attackers' strategies. To provide a realistic baseline, this thesis uses the EMBER dataset and a Gradient Boosted Decision Tree classifier as the target black-box detector. The GBDT model from the EMBER framework was chosen for its strong detection performance. Two different adversarial malware obfuscation architectures were evaluated: MalGAN and malware_rl. First, MalGAN is the implementation of a generative adversarial network. It is designed to generate obfuscated malware examples. It comprises three parts: a generator, a discriminator, and a black-box detector. The main idea is for the generator and the discriminator to go back and forth, with occasional black-box queries. This allows the generator to learn how to fool the black-box detector. On the other hand, the malware_rl framework is a reinforcement learning algorithm, which uses an agent to interact with the black-box model. The agent takes discrete actions to interact with the environment, including the detector, and receives outputs and rewards. These two approaches were chosen because they represent the foundations of their respective architectures. This allows for a faithful comparison of two different architectures. This thesis shows that MalGAN and the RL-based approach achieved high evasion rates. Recommendations are made to combine these two architectures into a single architecture by leveraging their strengths. |
