PhilaeX: Explaining the Failure and Success of AI Models in Malware Detection

Given a classifier $f(\mathbf{x})\rightarrow[0,1]^{|C|}$ to be explained, and its predictions of the probabilities of $|C|$ class labels for the input data sample $\mathbf{x}=(x_{1},x_{2},...,x_{m})\in R^{m}$ (in the features space), the data sample $\mathbf{x}\in R^{m}$ consists of $m$ features. For example, the suspicious Android app can be represented in the features space by the TF-IDF [Rajaraman and Ullman, 2011] values of its permissions [Arp et al., 2014]. The aim of the model explanation is to find the optimized features attribution vector $\mathbf{A}=(a_{1},a_{2},...,a_{m})\in R^{m}$ that quantitatively measure how the model to be explained $f(\mathbf{x})$ makes the prediction of the input sample’s class label according to each features contributions. That is, the optimization can be formally represented by:

where $g(\cdot)\in\mathcal{G}$ is the surrogate model to the original classifier $f(\cdot)$, which aims to mimic the predictions as $f(\mathbf{x})$ for the same sample $\mathbf{x}$, and the weights $w$ measures the joint contributions by the features in this surrogate model. The selection function $h(\mathbf{x})$ returns the optimized features set $\mathbf{x^{*}}$ that make the significant contributions to the model’s predictions, given sample $\mathbf{x}$. The attribution vector $\mathbf{A}$ is obtained only if the minimized difference between the surrogate model $g(\mathbf{x})$’s prediction and that of the original model $f(\mathbf{x})$ to be explained is obtained. Therefore, the choice of the surrogate model and the features $\mathbf{x^{*}}$ that their attributions are computed is critical to the explanation fidelity on the model’s prediction behavior on the sample $\mathbf{x}$.

In the remaining part of this section, we will introduce our proposed model explainer, PhilaeX, that starts the features attribution vector construction for the input sample $\mathbf{x}=(x_{1},x_{2},...,x_{m})\in R^{m}$ from an empty vector (i.e., Null). The whole construction process consists of two major stages: (1) The features selection strategy, i.e., $h(\mathbf{x})\in R^{n},n\leq m$, that picks up the features with the significant contributions towards the model’s prediction on $\mathbf{x}$ is selected; (2) The quantification of the contribution for the selected features through a Ridge regression that is the surrogate model to the original model $f(\mathbf{x})$.

The perturbation of the features values to obtain the synthetic input data samples $\mathbf{X^{{}^{\prime}}}$ in the training of the surrogate model $g(\mathbf{X^{{}^{\prime}}})$ may not work well in the cyber security field. In LIME [Ribeiro et al., 2016], the response of the model to the changes of the input variables is obtained by random perturbation of the input sample’s feature values in a small range. This can allow a fast preparation of a large amount of synthetic data to train the surrogate linear model and help the explainer to attribute the model’s behavior accordingly. However, it can also lead to the shortage of stable explanation that the features attribution values may vary a little among different times of explanation, given the same input sample. In addition, the perturbation strategy on features magnitude to generate the synthetic data to train the surrogate model may not work well in the cyber security field. For example, the normal way to camouflage the malware to evade the AI detector is to “add” a certain types of permission in the app, where the small perturbation of the features values is not impossible.

In PhilaeX, the features selection function $h(\mathbf{x})\in R^{n},n\leq m$ is to pick up the subset of the features from the input sample $\mathbf{x}$ that is optimized to describe (i.e., explain) the model’s prediction behavior. Specifically, there are two steps to obtain the candidate features for attribution, which are core features and features with positive individual contributions, respectively. The first step is to identify a set of core features $\mathbf{x_{c}}=\{x_{i}\in\mathbf{x}\}$ from the original sample $\mathbf{x}$, which are the base of the sample $\mathbf{x}$ that leads the model $f(\mathbf{x_{c}}):\to 0.5$ (i.e., the boarder line of the prediction). We assume that the model $f(\mathbf{x_{c}})$ make a “hesitated” decision for the sample with such core features only, where the model has around 50% confidence on its prediction of the sample’s class, and the actual prediction on the original input sample $f(\mathbf{x})$ is made by the joint contribution from both the core features and part of the remaining features.

In order to obtain the core features for the given sample $\mathbf{x}$, we start from an empty feature vector that contain no feature. The following steps are to find out the candidate core features in a recursive way, where the target is to find the subset of features that leads the model $f(\mathbf{x}_{c}):\to 0.5$ as close as possible (i.e., the local minimum of the $abs(f(\mathbf{x}_{c})-0.5)$. The detailed algorithm about core features identification are in Algorithm 1.

Once the core features $\mathbf{x}_{c}$ is obtained, we are looking for the features that can increase the prediction confidence of the model toward the prediction score on the original sample $\mathbf{x}$. Formally, we define the acquisition of such features with positive individual contributions, i.e., $\mathbf{x}_{p}$, as:

where the symbol “$+$” means the concatenation of two features vectors, i.e., $\mathbf{x}_{c}$ and $\mathbf{x}_{p}$. The candidate features set is initialized as $\mathbf{x}_{p}=\phi$. For every feature $x_{i}\in\mathbf{x}\backslash\mathbf{x}_{c}$ that is added into $\mathbf{x}_{p}$, the model’s prediction on $f(\mathbf{x}_{c}+\mathbf{x}_{p}:\to f(\mathbf{x})$.

The aim is to identify the features in the input sample $\mathbf{x}$ to enhance the confidence of the model significantly when it outputs the prediction of the input sample. Accordingly, those features that lead the model to the opposite of the prediction on the sample $\mathbf{x}$ will be ignored.

The features we picked up from the previous steps, i.e., the core features $\mathbf{x}_{c}$ and the features with positively individual contributions $\mathbf{x}_{p}$, form the set of the candidate features, where features attribution by PhilaeX will be applied. There are two reasons that we only attribute the subset of features in the input sample $\mathbf{x}$: (1) The features attribution on such features $\mathbf{x}_{s}=\mathbf{x}_{c}+\mathbf{x}_{p}$ allows the explainer to reveal the major reason that the model made the prediction on the original sample $\mathbf{x}$. As the discussion in Section 3.3, it is not always true for all features in the sample $\mathbf{x}$ that make positive significant contributions towards the model’s prediction of the class label on the sample. (2) The explanation on such subset of features will be more efficient that that on the all the features of the input sample $\mathbf{x}$.

The joint contributions made by the cooperation among these features are the necessary to form the complete quantitative explanation (i.e., features attribution) for the model $f(\mathbf{x})$, which have not yet been considered by the previous two steps in Section 3.2 and Section 3.3. In this step, we quantify each feature’s contribution to the model’s prediction by training a Ridge regression model $g(\cdot)$ as the surrogate model to the original model $f(\cdot)$, where the weights of each feature in the regression model are considered as the features attribution. The reason we use the Ridge regression as the surrogate model is for its simplicity, efficiency and its nature for estimating the coefficients (i.e., weights) where independent variables are highly correlated [Hilt and Seegrist, 1977].

Specifically, the weights $\mathbf{w}\in R^{\|\mathbf{x}_{s}\|}$ in Ridge regression can be estimated by the optimization of the following equation:

where the L2 regularization applies to reduce sensitivity to single feature and accordingly decrease the possibility of overfitting in the model training.

Finally the features attribution vector is defined as $\mathbf{A}=\mathbf{w}$ that considers both the individual contribution from each features and the joint contributions from the cooperation among these features $\mathbf{x}_{s}$.

We use two datasets in the evaluation on the explanation fidelity by PhilaeX. The first dataset, DREBIN [Arp et al., 2014], was used to test a lightweight Android malware detector, where the features of the suspicious Android apps were extracted from the application’s manifest file AndroidManifest.xml and disassembled dex code from the bytecode by the static analysis technique. The features that DREBIN extracted fall into 8 categories, like requested permissions, restricted API calls and network addresses, etc. In the DREBIN dataset, there are 5,560 Android malware apps and 123,453 non-malware apps in total. However, in our experiments, we randomly selected 5,555 malware samples and 5,555 non-malware apps, in order to build a balanced dataset for the model’s training. Further, the dis-joint training set and testing set used in the evaluation are built through a random split of these 11,110 samples, which generates a training set of 7,442 samples and a testing set of 3,668 samples. For each sample, the text features data in a sample will be converted into the features vector in the form of floating numbers. Specifically, all the features in the training dataset will be encoded by the tf-idf algorithm [Rajaraman and Ullman, 2011], that measures the importance of each feature in the dataset. The dimension of the features vector is 43,157, which is high dimension.

The second dataset used in the experiments is the PDF malware dataset [Smutz and Stavrou, 2012] that has 4,999 malicious samples and 5,000 benign samples. We use the 135 features suggested by [Guo et al., 2018], where the features have been encoded into binary (i.e., 0 or 1) values.

We test the explanation capability of PhilaeX for different AI models that cover the shallow (classical) machine learning and the recent emerging deep learning models. First, we trained a SVM [Arp et al., 2014] [Zhao et al., 2011] [Li et al., 2015] model, which is a classical shallow machine learning model and has been widely used as the classifier for binary classification tasks before the deep learning methods dominate this field. For a given sample in the feature space, SVM maps the relatively low dimension data into a high-dimension space such that the separation between two classes becomes more apparent, and thus is able to predict the sample’s class more accurately.

Specifically, we trained a SVM model with the Radial basis function (RBF) kernel [Vert et al., 2004], where the parameter that defines the inverse degree of the influence by a single training sample $\gamma$ is set to $1.0$. We trained two SVM classifiers for the Android malware detection task on the DREBIN dataset and the PDF malware detection task on the PDF malware dataset. In the remaining part of this section, we will use PhilaeX to explain the prediction behavior of these two classifiers (i.e., AI models).

In addition, we also trained a deep learning model for the Android malware detection task. BERT [Devlin et al., 2018], the transformer-based classifier that was proposed by Google for natural language processing (NLP) tasks in 2018, is used to classify the Android malware in the DREBIN dataset. We use the BERT implementation from HuggingFace Transformers library [Wolf et al., 2019] that is not sensitive to the letters case and the default parameters, such as the maximum length of text (128) and the learning rate (4e-5). There are 8 samples used in a single batch and 5 epochs were running in the training process of the BERT model. We trained a surrogate SVM model to the BERT Android malware detector in the model explanation, in order to avoid the complicated word embedding mechanism that converts the text tokens to numerical representations. Such surrogate SVM has highly similar prediction behavior as the BERT, given the sample input sample, where the TPR = 0.9984 and FPR = 0.0029.

Both the trained SVM and BERT models used in the Android malware detection tasks present good performance. The true positive rate (TPR) for both classifiers are around 0.96 with a 0.04 false positive rate (FPR). In addition, we also trained a separate SVM classifier and Random Forest classifier for the PDF malware detection task, which uses the default parameters.

We firstly evaluate the explanation capability of PhilaeX on how the adversarial samples of Android malware evade the trained malware detector (that was with high TPR and low FPR on DREBIN dataset) in quantitative way. In the evasion attack, we assume the attacker has full knowledge of the features space and access to the model’s prediction score. That is, the attacker is able to manipulate the data sample, which class is to be predicted by the SVM or BERT classifier, such as adding the features in the sample.

In this experiment, we only add (i.e., activate) the “permission” features to the existing sample’s feature vector of Android malware, because such addition operation will not change the functionality of the original malware [Liu et al., 2019]. One adversarial sample is generated by Genetic Algorithm that is extended from [Liu et al., 2019] and the optimised set of “permission” features is selected to help the original sample bypasses the malware detection by the classifier. Specifically, in the Genetic Algorithm, the fitness value is defined by the model’s prediction score towards the non-malware class of the candidate adversarial sample. The convergence of the algorithm is fulfilled if (1) the Genetic Algorithm that has been running for 500 loops that has a high possibility to make the evasion attack by the adversarial samples successful; (2) the prediction score towards the non-malware class stay the same at a high level for at least 10 times; or (1) the fitness value is larger than 0.99 which implies the model has extremely high confidence on its incorrect prediction for the adversarial sample. In total, there are 200 malware samples from the testing set randomly selected as the seeds to generate the adversarial samples. The adversarial samples dataset used in the explanation for SVM has 499 samples. In the explanation for BERT, there are dis-joint 500 samples used.

The aim of the evaluation is to observe the capability of the model explanation by PhilaeX in terms of the percentage of “good” explanations. An adversarial sample has a “good explanation”, only if a certain number of the activated features in this sample are attributed with positive values. A high number of “activated features” are identified in terms of their attribution values and means that the model explanation verifies the assumption that the model is evaded because of the activated features in the adversarial sample.

In the experiment, we compare the explanation capability of PhilaeX against LIME [Ribeiro et al., 2016], SHAP [Lundberg and Lee, 2017] and MPT explainer [Lu and Thing, 2021]. The reasons that we use these three explainable AI methods as the baseline are: (1) LIME is a popular explainable AI method that explains the models by learning a linear surrogate model. (2) The explanation generated by SHAP is based on the computation of Shapley value [Roth, 1988], which concept has been widely used in cooperative game theory. (3) The recently MPT explainer is based on the modern portfolio theory [Markowitz, 1952] that was proposed in economics to allocate the investment to different assets for a maximum return with minimum risk. In the evaluation, we vary the threshold of the “good explanation” from 0% activated features in the adversarial samples identified to 90% activated features identified. This allows us to observe the robustness of the explanations from different explainable AI methods. In Fig 1, it shows PhilaeX can identify more activated features from the adversarial samples compared to LIME, MPT explainer and SHAP, when the same threshold of “good explanation” is used and the threshold value is less than 40% in SVM and 20% in BERT. In addition, PhilaeX’s explanation shows much robustness that is verified by the slower decreasing curve, compared to SHAP and MPT explainer. This conclusion still holds true when we compare the robustness of PhilaeX and LIME, considering the unstable explanation in LIME that is caused by the random perturbation on the features’ values.

In the explanation of BERT, PhilaeX shows slight lower ratio of “good explanations”, when the threshold of “good explanation” is less than 30%. This is possibly because BERT considers more joint contributions among the features that reduces the effect by single features accordingly. However, we see that PhilaeX still presents a relatively robust explanation capability among these explainable AI methods, because of its slower curve decline.

In the fidelity test, the aim is to evaluate if the explainer attributes high values for the features that has high impact on the model’s prediction behavior. Specifically, there are two kinds of tests we used in the experiments: (1) Deduction test that removes a certain number of features with high attribution values will lead the model to predict the manipulated sample as the opposite class. That is, the less such high attribution value features are removed, the higher the explanation fidelity. For example, the SVM model predicts a manipulated sample of malware as non-malware, if the feature with the top attribution value is removed. This means that this feature is correctly attributed in the explanation; (2) In Augmentation test, we activate a certain number of features in a non-malware sample. These features are from a malware sample and are attributed with high attribution values in the model explanation on this malware sample. It is expected that the model’s prediction on the manipulated non-malware sample as malware, if the explanation is correct. That is, the correctly attributed features in a malware sample may have strong individual impact on the model’s prediction behavior that lead the model towards the malware class.

We use the positive classification rate (PCR) [Guo et al., 2018] as the evaluation metric to quantify the fidelity of the explanations. The PCR is defined as the ratio of samples which retains their original class after the manipulation through deduction or augmentation. The PCR in an explanation with high fidelity will be as low as possible through a deduction test, and as high as possible by the augmentation test.

In this experiment, we test the explanation fidelity by PhilaeX, when it is used to explain the Random Forest and SVM classifiers on the PDF malware detection task. In Fig. 2(a) and Fig. 2(b), we can observe that for both RF and SVM, PhilaeX has a significant higher fidelity explanation than that of MPT explainer, which are measured by the lower PCRs. This finding verifies that the features selection function $h(\mathbf{x})$ in Section 3 guarantees the following features attribution to assign high attribution values to the important features. In addition, the high fidelity (in terms of PCR) is stable although the number of features used is increasing. This means that PhilaeX is more capable of identifying the important features (by attributing it with higher value) than that of MPT explainer.

In Fig. 2(c) and Fig. 2(d), the features with high attribution values by PhilaeX will generally guarantee a high PCR for both RF and SVM, when the number of features used are small. However, the PCRs for PhilaeX are getting lower than that of MPT explainer when around 50 and more features are used in the augmentation test. This is probably due to the joint contribution by all the features becoming stronger as the number of features used increases.

The average running time to explain the SVM’s prediction on a single data sample of Android malware apps is around 6.37 seconds, compared to the MPT explainer with around 15.44 seconds. This is probably due to the efficient the optimization process of Ridge regression.