Shapley Sets: Feature Attribution via Recursive Function Decomposition

In co-operative game theory, one central question is that of fair division: if players form a coalition to achieve a common goal, how should they split the profits? Let $N$ be the set $\{1,2,...n\}$ of players and $2^{N}$ all coalitions of players. A function $v:2^{N}\rightarrow\mathbb{R}$ is the $n$-person game in characteristic form, such that $v(S),S\subseteq N$ defines the worth of coalition $S$ where $v(\varnothing)=0$. A solution concept is a mapping assigning a vector $\mathbf{x}\in\mathbb{R}^{n}$ to the game $v$. The Shapley value [1] is the most widely known solution concept which uniquely satisfies certain axioms of fairness: efficiency, dummy, symmetry and additivity. Please see [1] for definitions.

Under efficiency, the Shapley value decomposes the value of the grand coalition $v(N)-v(\varnothing)$ to attribute worth to each individual player. The Shapley value is a fully separable function (Definition 1.2) such that $v(N)-v(\varnothing)=\sum_{i}^{n}\phi_{i}(v)$.

Specifically, the function $f$ is also called fully additively separable if $k=n$, while it is regarded as fully non-separable if $k=1$. While there are other forms of separability, In this paper we use the term separable to refer to additive separability.

The set function $v$ may not be fully separable. Within coalitional games this is due to the interaction between players. Consider the following example for the game $v$ with player set $N=\{1,2,3\}$ and $v(1)=1,v(2)=0,v(3)=0,v(1,2)=1,v(1,3)=1,v(2,3)=2,v(1,2,3)=3$. Clearly the game is not fully separable as $v(1)+v(2)+v(3)\not=v(1,2,3)$. The non-separable interaction effects within coalitional games are dealt with by solution concepts which map partially separable into fully separable functions, allowing an individual attribution of worth to each player. The Shapley value provides an attribution where each player receives an average of their marginal contribution to all coalitions.

The problems with Shapley value attributions discussed above occur as it assigns individual value to variables belonging to Non-Separable Variable Groups (NSVGs) in regards to the underlying partially separable function $f$ (Definition 1.2). Non-separable groups are used to describe the formed variable groups $\{\mathbf{X}_{1},...,\mathbf{X}_{k}\}$ after a complete (or ideal) decomposition of $f$. An NSVG can also be defined as the minimal set of all interacted variables given the function $f$ which we explicate in Definition 2.1.

Translating Definition 2.1 for feature attribution, given that $f(\mathbf{x})_{\mathbf{X}_{S}}$ is a function over the domain of all the possible subsets of $\mathbf{X}_{S}\subseteq\mathbf{X}$ we can rewrite Equation 5 in terms of $v(\mathbf{x},\mathbf{X}_{S})$, where $v$ could represent any of the value functions from the previous section but in this paper we restrict $v\in\{v_{cond},v_{bs}\}$. By setting $\mathbf{X}_{i}=\{X_{i}\}$ and $\mathbf{X}_{j}=\mathbf{X}_{S}$, for $v_{bs}$, given that $|\mathbf{X}_{S}|$ is minimised, if there exist any candidate vectors $\mathbf{x},\mathbf{x^{\prime}}$ such that

then $\{X_{i}\}\cup\mathbf{X}_{S}$ is a NSVG.

For $v_{cond}$, given that $|\mathbf{X}_{S}|$ is minimised, if there exists any candidate vector $\mathbf{x}$ such that

then $\{X_{i}\}\cup\mathbf{X}_{S}$ is a NSVG.

Given the partially separable function from Example 2, under $v_{bs}$, $\{X_{2},X_{3}\}$ is a NSVG as $v_{bs}(\mathbf{x},\mathbf{x^{\prime}},\{X_{3},X_{2}\})-v_{bs}(\mathbf{x},\mathbf{x^{\prime}},\{X_{2}\})\not=v_{bs}(\mathbf{x},\mathbf{x^{\prime}},\{X_{3}\})$ for settings $\mathbf{x}=(1,1,1)$ and $\mathbf{x^{\prime}}=(0,0,0)$.

Given the partially separable function from Example 1, under $v_{cond}$, the set $\{X_{2},X_{3}\}$ is a NSVG as $v_{cond}(\mathbf{x},\{X_{3},X_{2}\})-v_{cond}(\mathbf{x},\{X_{2}\})\not=v_{cond}(\mathbf{x},X_{3})$ for setting $\mathbf{x}=(1,1,1)$.

In this paper, we propose an alternative attribution method which, unlike the Shapley value, does not separate NSVGs to assign attribution. We work under the intuition that any interacting feature whether that be in the model or in the data should not be considered as separate players in the coalitional game but should be awarded value together. In both the examples above, $X_{2}$ and $X_{3}$ would receive joint attribution under our proposed method.

Given the partially separable function $f$ satisfying Definition 1.2, variable set $\mathbf{X}=\{X_{1},X_{2},...,X_{n}\}$, and a specified value function $v_{cond,marg}(\mathbf{x},\mathbf{X}_{S})$, our proposed solution concept $\varphi$, which we term Shapley Sets (SS), finds the optimal decomposition of $f$ into the set of $m>1$ NSVGs $\{\mathbf{X}_{1},...,\mathbf{X}_{m}\}$. The resulting variable grouping $\{\mathbf{X}_{1},...,\mathbf{X}_{m}\}$ satisfies Definition 1.2 and each variable group is composed solely of variables $\mathbf{X}_{i}$ which satisfy definition 2.1. From Definition 1.2, $f(\mathbf{x})=\sum_{i=1}^{m}v(\mathbf{x},\mathbf{X}_{i})$. Given a prediction to be attributed, $f(\mathbf{x})$, our proposed attribution, $\varphi$, therefore returns the attribution for each variable group $\mathbf{X}_{i},\forall i\in m$ given as:

Proposition 2.2 shows how the attribution given by Shapley Sets (SS) to variable $X_{i}$, $\varphi_{\mathbf{X}_{i}}(v,\mathbf{x})$, is equivalent to the Shapley value when played over the feature set $\mathbf{Z}$ containing the set of NSVGs $\{Z_{1},...,Z_{m}\}=\{\mathbf{X}_{1},...,\mathbf{X}_{m}\}$ for a given $v\in\{v_{cond},v_{bs}\}$. SS therefore satisfies the same axioms of fairness as the Shapley value: efficiency, dummy, additivity and symmetry when played over this feature set. However, we have discussed how, despite its axioms, the Shapley value can generate misleading attributions in the presence of feature interaction. In Section 4 we therefore give practical advantages of the SS over the Shapley value. First however, we provide a method for finding the optimal decomposition of $f$ into its NSVGs.

Determining the NSVGs of a function $f$ could be achieved manually by partitioning the variable set and determining interaction over every possible candidate vector. However, this would be computationally intractable. Instead, there exists a large body of literature surrounding function decomposition in global optimization problems. Of this work, automatic decomposition methods identify NSVGs. We therefore propose a method for calculating SS which is based on the Recursive Decomposition Grouping algorithm (RDG) as introduced in [12].

To identify whether two sets of variables $\mathbf{X}_{i}$ and $\mathbf{X}_{j}$ interact, RDG uses a fitness measure, based on Definition 2.1 with candidate vectors, $\mathbf{x},\mathbf{x}^{\prime}$ as the lower and upper bounds of the domain of $\mathbf{X}$. If the difference between the left and right hand side of Equation 5 meets some threshold $\epsilon=\alpha\min\{|f(\mathbf{x}_{1})|,...,|f(\mathbf{x}_{k})|\}$ where $\mathbf{x}_{k}$ is a randomly selected candidate vector, then $\mathbf{X}_{i}$ and $\mathbf{X}_{j}$ are deemed by RDG to interact. To adapt RDG for $v_{cond}$ and $v_{bs}$, we propose an alternative fitness measure, Definition 3.1, with candidate vectors $\mathbf{x},\mathbf{x^{\prime}}$ randomly sampled from $\mathbf{X}_{input}$ which can identify NSVGs in the function and/or in the model.

We substitute the SS fitness measure into the RDG algorithm which identifies NSVGs by recursively identifying individual variable sets $\mathbf{X}_{j}$ with which a given variable $X_{i}$ interacts with. If $X_{i}$ and a single variable $X_{j}$ are said to interact they are placed into the same NSVG, $\mathbf{X}_{1}$. At which point conditional interaction between $\mathbf{X}_{1}$ and the remaining variables is identified. The algorithm iterates over every variable $X_{i}\in\mathbf{X}$ and returns the set of NSVGs. To compute the SS attributions for a given prediction $f(\mathbf{x})$ we compute $v(\mathbf{x},\mathbf{X}_{i})$ for each NSVG, $\mathbf{X}_{i}$. Our full algorithm is shown in Algorithm 2. The runtime of SS is $O(nlogn)$ as proven in [12].

The selection of the value function $v$ determines the variable grouping generated. Used with $v_{bs}$, as interacting features are placed in the same NSVG, the attributions resulting from SS will be more faithful to the underlying model. The SS attribution for $v_{bs}$ in Example 2 would be $\varphi_{X_{1}}=1$ and $\varphi_{X_{2},X_{3}}=2$.

Used with $v_{cond}$, as interacting features are placed in the same NSVG, the attributions resulting from SS do not suffer from the violation of sensitivity as described in Section 1.2. Consider again Example 1, as $X_{2},X_{3}$ now belong to a NSVG, the SS attributions for $X_{1},X_{3}$ are now equal across both $f$ and $f_{2}$ therefore robust to whether non-directly impacting features are included in the model. SS offer a further advantage when used to compare the attributions under on and off-manifold examples. Consider again Example 2 yet now with $X_{1}=\alpha X_{2}$. The SS attribution via $v_{marg}$ would be $\varphi_{X_{1}}=1$ and $\varphi_{X_{2},X_{3}}=2$. However, if SS was calculated via $v_{cond}$ $\varphi_{\{X_{1},X_{2},X_{3}\}}=3-\mathbb{E}[f(\mathbf{X})]$ indicating that $f$ is non-separable and all the features interact.

The comparison between on and off-manifold SS therefore indicate where the feature interaction takes place. We have thus far provided an alternative attribution method to the Shapley Value, SS which can be computed in $O(nlogn)$ time with $n$ being the number of features. SS can be adapted for arbitrary value functions and offers several advantages over Shapley value based attributions when used with on and off-manifold value functions. In Section 6 we empirically validate the theoretical claims made above but first we discuss related work.

As Shapley value based feature attribution has a rich literature, we differentiate SS from three approaches which are closest in essence to ours. SS enforce a coalition structure on the Shapley value such that players cannot be considered in isolation from their coalitions. The Owen value is a solution concept for games with an existing coalition structure [13]. The Owen value is the result of a two–step procedure: first, the coalitions play a quotient game among themselves, and each coalition receives a payoff which, in turn, is shared among its individual players in an internal game. Both payoffs are given by applying the Shapley value. This approach is not equivalent to SS, who assume no prior coalitional structure, and instead finds the optimum coalition structure which is the decomposition of $v$ into its NSVGs.

Shapley Residuals [11] capture the level to which a value function is inessential to a coalition of features. They show for $v_{cond},v_{marg}$, that if the game function can be decomposed into $v(\mathbf{x},\mathbf{X}_{S})=v(\mathbf{X}_{T})+v(\mathbf{X}_{\bar{T}})$ for $\mathbf{X}_{T}\subset\mathbf{X}_{S}$ then the value function $v$ is inessential with respect to the coalition $\mathbf{X}_{T}$. In this way we can view Shapley residuals, $r_{S}\neq 0$ as an indication that a coalition is an non-separable variable group. However, the Shapley residuals are built on complex Hodge decomposition [14], are difficult to understand and do not offer a better way of attributing to features. In contrast, SS is built on the idea of additive separability, easier to understand, less computationally expensive and propose a solution to the issues with the Shapley value which are analogous to those Shapley residuals were designed to identify.

Grouped Shapley Values Determining the Shapley value of grouped variables has been previously suggested in [15, 16] which identify interaction in the data (based on measures of correlation) to then partition the features into groups, after which the Shapley value is then calculated. Shapley Sets is distinct from the above approaches in the following ways. Firstly, Shapley Sets is capable of uncovering interaction in the model as well as in the data. Secondly, Shapley Sets is designed to find the optimal grouping of the features such that the Shapley value theoretically reduces to the simple computation in Equation 8. Therefore, the grouping under Shapley Sets requires linear time to compute (given the prior decomposition of the variable set under log linear time), whereas the grouping proposed under grouped shapley values [15, 16] still requires exponential computation (to compute exactly although this can be approximated). Shapley Sets, to our knowledge, is the first contribution to the feature attribution literature which automatically decomposes a function into the optimal variable set by which to award attribution.

We begin with two synthetic experiments. The first of these motivates the use of SS in the presence of interaction in the model. The second motivates the use of the SS in the presence of interaction in the data. We then compare SS to existing Shapley value (SV) based attribution methods on three benchmark datasets. We first however, outline how the value functions $v_{bs},v_{cond}$ are computed for our experiments.

As discussed above, $v_{bs}$ takes as input arbitrary reference vectors. For our experiments we select $v_{marg}$ such that $v_{bs}=v_{marg}$ (Equation 2). The expectation is taken over the empirical input distribution $\mathbf{X}_{input}$ For the calculation of $v_{cond}$ (Equation 4), as the true conditional probabilities for the underlying data distribution are unknown we approximate $p(\mathbf{X}_{\bar{S}}|\mathbf{X}_{S}=\mathbf{x}_{s)}$ using the underlying data distribution. Approximating conditional distributions can be achieved by directly sampling from the empirical data distribution. However, as noted in [3], the this method of approximating $p(\mathbf{X}_{\bar{S}}|\mathbf{X}_{S})=\mathbf{x}_{s}$ suffers when $|\mathbf{X}_{S}|>2$, due to sparsity in the underlying empirical distribution. We therefore adopt the approach of [3], where under the assumption that each $\mathbf{x}\in\mathbf{X}$ is sampled from a multivariate Gaussian with mean vector $\mathbf{\mu}$ and covariance matrix $\mathbf{\Sigma}$, the conditional distribution $p(\mathbf{X}_{\bar{S}}|\mathbf{X}_{S}$ is also multivariate Gaussian such that $p(\mathbf{X}_{\bar{S}}|\mathbf{X}_{S}=\mathbf{x}_{s})=\mathcal{N}_{\bar{S}}(\boldsymbol{\mu}_{\bar{S}|S},\mathbf{\Sigma}_{\bar{S}|S})$ where $\boldsymbol{\mu}_{\bar{S}|S}=\boldsymbol{\mu}_{\bar{S}}+\mathbf{\Sigma}_{\bar{S}S}\mathbf{\Sigma}^{-1}_{SS}(\mathbf{x}_{S}-\boldsymbol{\mu}_{S})$ and $\mathbf{\Sigma}_{\bar{S}|S}=\mathbf{\Sigma}_{\bar{S}\bar{S}}+\mathbf{\Sigma}_{\bar{S}S}\mathbf{\Sigma}^{-1}_{SS}\mathbf{\Sigma}_{S\bar{S}}$. We can therefore sample from the conditional Gaussian distribution with expectation vector and covariance matrix given by $\boldsymbol{\mu}_{\bar{S}|S}$ and $\mathbf{\Sigma}_{\bar{S}|S}$ where $\boldsymbol{\mu}$ and $\mathbf{\Sigma}$ are estimated by the sample mean and covariance matrix of $\mathbf{X}_{input}$.

This paper has introduced Shapley Sets (SS), a novel method for feature attribution, which automatically and optimally decomposes a function $f$ into a set of NSVGs by which to award attribution. We have shown how SS generates more faithful explanations in the presence of feature interaction both in the data and in the model than Shapley value-based alternatives. To our knowledge, SS is the only method in the literature which automatically generates a grouped attribution vector. Below we explore some limitations of SS and ideas for future work. Sensitivity to Parametrisation: In Algorithm 2, $\epsilon$ determines the degree to which two sets of variables are considered interacting. The original RDG algorithms recommends the setting $\epsilon$ as proportional the magnitude of the objective space. This setting works well for SS Interventional. However, we noticed a large variation in the variable grouping generated by SS Conditional under this setting of $\epsilon$. This is not surprising as it is known that $v_{cond}$ is sensitive to feature correlations in the data and it is difficult to know how much correlational structure to allow before two features are considered to be causally linked. Future work should therefore look at alternative methods of function decomposition which are not so dependent on the parametrisation of $\epsilon$ [19]. Assumption of Partially-Separable Model SS assumes that the model to be explained is partially separable. If we consider the function $f(\mathbf{X})=X_{1}X_{2}X_{3}$, SS would result in a single attribution to all three features of $f(\mathbf{x})$. This is not useful from an explanation perspective although does inform us about the nature of the underlying model. Furthermore, the assumption of a partially separable function is also made by the Shapley value [2]. Future work should consider function decomposition under a wider class of separability such as multiplicative separability where associated algorithms decompose a function into its additive and multiplicative separable variable sets [19].

We would like to thank Giulia Occhini, Alexis Monks, Isobel Shaw, Jennifer Yates, for their invaluable support during the writing of this paper. This work was supported by an Alan Turing Institute PhD Studentship funded under EPSRC grant EP/N510129/1.