S-LIME: Stabilized-LIME for Model Explanation

Data Mining and machine learning models have been widely deployed for decision making in many fields, including criminal justice (Zeng et al., 2015) and healthcare (Rajkomar et al., 2018; Miotto et al., 2018). However, many models act as “black boxes” in that they only provide predictions but with little guidance for humans to understand the process. It has been a desiderata to develop approaches for understanding these complex models, which can help increase user trust (Ribeiro et al., 2016), assess fairness and privacy (Angwin et al., 2016; Doshi-Velez and Kim, 2017), debug models (Koh and Liang, 2017) and even for regulation purposes (Goodman and Flaxman, 2017).

Model explanation methods can be roughly divided into two categories (Du et al., 2019; Wang et al., 2020): intrinsic explanations and post hoc explanations. Models with intrinsically explainable structures include linear models, decision trees (Breiman et al., 1984), generalized additive models (Hastie and Tibshirani, 1990), to name a few. Due to complexity constraints, these models are usually not powerful enough for modern tasks involving heterogeneous features and enormous numbers of samples.

Post hoc explanations, on the other hand, provide insights after a model is trained. These explanations can be either model-specific, which are typically limited to specific model classes, such as split improvement for tree-based methods (Zhou and Hooker, 2019) and saliency maps for convolutional networks (Simonyan et al., 2013); or model-agnostic that do not require any knowledge of the internal structure of the model being examined, where the analysis is often conducted by evaluating model predictions on a set of perturbed input data. LIME (Ribeiro et al., 2016) and SHAP (Lundberg and Lee, 2017) are two of the most popular model-agnostic explanation methods.

Researchers have been aware of some drawbacks for post hoc model explanation. (Hooker and Mentch, 2019) showed that widely used permutation importance can produce diagnostics that are highly misleading due to extrapolation. (Ghorbani et al., 2019) demonstrated how to generate adversarial perturbations that produce perceptively indistinguishable inputs with the same predicted label, yet have very different interpretations. (Aïvodji et al., 2019) showed that explanation algorithms can be exploited to systematically rationalize decisions taken by an unfair black-box model. (Rudin, 2019) argued against using post hoc explanations as these methods can provide explanations that are not faithful to what the original model computes.

In this paper, we focus on post hoc explanations based on perturbations (Ribeiro et al., 2016): one of the most popular paradigm for designing model explanation methods. We argue that the most important property of any explanation technique is stability or reproducibility: repeated runs of the explanation algorithm under the same conditions should ideally yield the same results. Unstable explanations provide little insight to users as how the model actually works and are considered unreliable. Unfortunately, LIME is not always stable. (Zhang et al., 2019) separated and investigated sources of instability in LIME. (Visani et al., 2020) highlighted a trade-off between explanation’s stability and adherence and propose a framework to maximise stability. (Lee et al., 2019) improved the sensitivity of LIME by averaging multiple output weights for individual images.

We propose a hypothesis testing framework based on a central limit theorem for determining the number of perturbation samples required to guarantee stability of the resulting explanation. Briefly, LIME works by generating perturbations of a given instance and learning a sparse linear explanation, where the sparsity is usually achieved by selecting top features via LASSO (Tibshirani, 1996). LASSO is known to exhibit early occurrence of false discoveries (Meinshausen and Bühlmann, 2010; Su et al., 2017) which, combined with the randomness introduced in the sampling procedure, results in practically-significant levels of instability. We carefully analyze the Least Angle Regression (LARS) (Efron et al., 2004) for generating the LASSO path and quantify the aymptotics for the statistics involved in selecting the next variable. Based on a hypothesis testing procedure, we design a new algorithm call S-LIME (Stabilized-LIME) which can automatically and adaptively determine the number of perturbations needed to guarantee a stable explanation.

In the following, we review relevant background on LIME and LASSO along with their instability in Section 2. Section 3 statistically analyzes the asymptotic distribution of the statistics which is at the heart of variable selection in LASSO. Our algorithm S-LIME is introduced in Section 4. Section 5 presents empirical studies on both simulated and real world data sets. We conclude in Section 6 with some discussions.

In this section, we review the general framework for constructing post hoc explanations based on perturbations using Local Interpretable Model-agnostic Explanations (LIME) (Ribeiro et al., 2016). We then briefly discuss LARS and LASSO, which are the internal solvers for LIME to achieve the purpose of feature selection. We illustrate LIME’s instability with toy experiments.

Consider at any given step when LARS needs to choose a new variable to enter the model. With sample size of $n$, let the current residuals be given by $\bm{r}=(r_{1},r_{2},\ldots,r_{n})$, and two candidate variables being $\bm{x_{\cdot i}}=(x_{1i},x_{2i},\ldots,x_{ni})$ and $\bm{x_{\cdot j}}=(x_{1j},x_{2j},\ldots,x_{nj})$ where we assume the predictors have been standardized to have zero mean and unit norm. LARS chooses the predictor that has the highest (absolute) correlation with the residuals to enter the model. Equivalently, one needs to compare $\hat{c}_{1}=\frac{1}{n}\sum_{t=1}^{n}r_{t}x_{ti}$ with $\hat{c}_{2}=\frac{1}{n}\sum_{t=1}^{n}r_{t}x_{tj}$. We use $\hat{c}_{1}$ and $\hat{c}_{2}$ to emphasize these are finite sample estimates, and our purpose is to obtain the probability that their order would be different if the query points were regenerated. To that end, we introduce uppercase symbols $\bm{R},\bm{X_{\cdot i}},\bm{X_{\cdot j}}$ to denote the corresponding random variables of the residuals and two covariates; these are distributed according to the current value of the coefficients in the LASSO path and we seek to generate enough data to return the same ordering as the expected values $c_{1}=E(\bm{R}\cdot\bm{X_{\cdot i}})$ and $c_{2}=E(\bm{R}\cdot\bm{X_{\cdot j}})$ with high probability. Our algorithm is based on pairwise comparisons between candidate features; we therefore consider the decision between two covariates in this section, and extensions to more general cases involving multiple pairwise comparisons will be discussed in Section 4.

By the multivariate Central Limit Theorem (CLT), we have

where

Without loss of generality we assume $\hat{c}_{1}>\hat{c}_{2}>0$. In general if the correlation is negative, we can simply negate the corresponding feature values for all the calculations involved in this section. Let $\hat{\Delta}_{n}=\hat{c}_{1}-\hat{c}_{2}$ and $\Delta_{n}=c_{1}-c_{2}$. Consider function $f(a_{1},a_{2})=a_{1}-a_{2}$. Delta method implies that

Or approximately,

where the variance estimates are estimated from the empirical covariance of the values $r_{t}x_{ti}$ and $r_{t}x_{tj}$, $t=1,\ldots,n$.

In similar spirits of (Zhou et al., 2018), we assess the probability that $\hat{\Delta}_{n}>0$ will still hold in a repeated experiment. Assume we have another independently generated data set denoted by $\{r_{t}^{*},x_{ti}^{*},x_{tj}^{*}\}_{t=1}^{n}$. It follows from (3) that

which leads to the approximation that

In order to control $P(\hat{\Delta}_{n}^{*}>0)$ at a confidence level $1-\alpha$, we need

where $Z_{\alpha}$ is the $(1-\alpha)$-quantile of a standard normal distribution.

For a fixed confidence level $\alpha$ and $n$, suppose we get the corresponding $p$-value $p_{n}>\alpha$. From (4) we have

This implied we would need approximately $n^{\prime}$ samples to get a significant result where

Based on the theoretical analysis developed in Section 3, we can run LIME equipped with hypothesis testing at each step when a new variable enters. If the testing result is significant, we continue to the next step; otherwise it indicates that the current sample size of perturbations is not large enough. We thus generate more synthetic data according to Equation (5) and restart the whole process. Note that we view any intermediate step as conditioned on previous obtained estimates of $\hat{\beta}$. A high level sketch of the algorithm is presented below in Algorithm 3.

In practice, we may need to set an upper bound on the number of synthetic samples generated (denoted by $n_{max}$), such that whenever the new $n^{\prime}$ is greater than $n_{max}$, we’ll simply set $n=n_{max}$ and go though the outer while loop one last time without testing at each step. This can prevent the algorithm from running too long and wasting computation resources in cases where two competing features are equally important in a local neighborhood; for example, if the black box model is indeed locally linear with equal coefficients for two predictors.

We note several other possible variations of the Algorithm 3.

Multiple testing. So far we’ve only considered comparing a pair of competing features (the top two). But when choosing the next predictor to enter the model at step $m$ (with $m-1$ active features), there are $p-m+1$ candidate features. We can modify the procedure to select the best feature among all the remaining candidates, by conducting pairwise comparisons between the feature with largest correlation ($\hat{c}_{1}$) against the rest ($\hat{c}_{2},\ldots,\hat{c}_{p-m+1}$). This is a multiple comparisons problem, and one can use an idea analogous to Bonferroni correction. Mathematically:

• •

  Test the hypothesis $H_{i,0}:\hat{c}_{1}\leq\hat{c}_{i}$, $i=2,\ldots,p-m+1$. Obtain $p$-values $p_{2},\ldots,p_{p-m+1}$.

• •

  Reject the null hypothesis if $\sum_{i=2}^{p-m+1}p_{i}<\alpha$.

Although straightforward, this Bonferroni-like correction ignores much of the correlation among these statistics and will result in a conservative estimate. In the experiments, we only conduct hypothesis testing for top two features without resorting to multiple testing, as it is more efficient and empirically we do not observe any performance degradation.

Efficiency. Several modifications can be made to improve the efficiency of Algorithm 3. At each step when $n$ is increased to $n^{\prime}$, we can reuse the existing synthetic samples and only generate additional $n^{\prime}-n$ perturbation points. One may also note that whenever the outer while loop restarts, we conduct repetitive testings for the first several variables entering the model. To achieve better efficiency, each new run can condition on previous runs: if a variable enters the LASSO path in the same order as before and has been tested with significant statistics, no additional testing is needed. Hypothesis testing is only invoked when we select more features than previous runs, or in some rare cases, the current iteration disagrees with previous results. In our experiments, we do not implement the conditioning step for implementation simplicity, as we find the efficiency gain is marginal when selecting a moderate size of features.

Rather than performing a broad-scale analysis, we look at several specific cases as illustrations to show the effectiveness of S-LIME in generating stabilized model explanations. Scikit-learn (Pedregosa et al., 2011) is used for building black box models. Code for replicating our experiments is available at https://github.com/ZhengzeZhou/slime.

An important property for model explanation methods is stability: repeated runs of the algorithm on the same object should output consistent results. In this paper, we show that post hoc explanations based on perturbations, such as LIME, are not stable due to the randomness introduced in generating synthetic samples. Our proposed algorithm S-LIME is based on a hypothesis testing framework and can automatically and adaptively determine the appropriate number of perturbations required to guarantee stability.

The idea behind S-LIME is similar to (Zhou et al., 2018) which tackles the problem of building stable approximation trees in model distillation. In the area of online learning, (Domingos and Hulten, 2000) uses Hoeffding bounds (Hoeffding, 1994) to guarantee correct choice of splits in a decision tree by comparing two best attributes. We should mention that S-LIME is not restricted to LASSO as its feature selection mechanism. In fact, to produce a ranking of explanatory variables, one can use any sequential procedures which build a model by sequentially adding or removing variables based upon some criterion, such as forward-stepwise or backward-stepwise selection (Friedman et al., 2001). All of these methods can be stabilized by a similar hypothesis testing framework like S-LIME.

There are several works closely related to ours. (Zhang et al., 2019) identifies three sources of uncertainty in LIME: sampling variance, sensitivity to choice of parameters and variability in the black box model. We aim to control the first source of variability as the other two depend on specific design choices of the practitioners. (Visani et al., 2020) highlight a trade-off between explanation’s stability and adherence. Their approach is to select a suitable kernel width for the proximity measure, but it does not improve stability given any kernel width. In (Zafar and Khan, 2019), the authors design a deterministic version of LIME by only looking at existing training data through hierarchical clustering without resorting to synthetic samples. However, the number of samples in a dataset will affect the quality of clusters and a lack of nearby points poses additional challenges; this strategy also relies of having access to the training data. Most recently, (Slack et al., 2020b) develop a set of tools for analyzing explanation uncertainty in a Bayesian framework for LIME. Our method can be viewed as a frequentist counterpart without the need to choose priors and evaluate a posterior distribution.

Another line of work concerns adversarial attacks to LIME. (Slack et al., 2020a) propose a scaffolding technique to hide the biases of any given classifier by building adversarial classifiers to detect perturbed instances. Later, (Saito et al., 2020) utilize a generative adversarial network to sample more realistic synthetic data for making LIME more robust to adversarial attacks. The technique we developed in this work is orthogonal to these directions, as . We also plan to explore other data generating procedures which can help with stability.