XAI Handbook:
Towards a Unified Framework for Explainable AI

Most definitions treat “explanation” and “interpretation” in a similar fashion, sometimes even as synonyms. However, there are subtle but fundamental differences that allow some initial distinction to be drawn between them. First, we need to ask what kind of semantic entity an explanation is: is it an action, an outcome, a process or an object? For most textbook definitions (see Table 2), explanations are seen as statements. In turn, such statements are nothing but descriptions about an already existing entity (the explanandum or the one which is subject to description). From a functional perspective, an explanation is therefore the process by which an explanandum is described. Finally, to avoid confusions between the process of explaining and its output, we refer to the former as explanation and the latter as the explanans.

To prevent any kind of circular definition, the explanandum needs to exist axiomatically, thus it has to refer to objects or symbols that are self-evidently true. In other words, we require explanandums to be factual and axiomatic.

As we seek explanations for AI models, we find such suitable facts in the form of low-level mathematical primitives used to build the models themselves: support vector machines have a decision boundary equation, coordinates of the support vectors, the tolerance threshold, etc. A Neural Network has values for each individual parameter, the equations governing how they connect with each other, the value of the cost function when computed on a particular input, the gradient that can be computed on that loss, etc. All of these are concrete, undisputed facts (assuming there are no bugs) that are suitable explanandums.

In a simplified, more intuitive form, a first definition of “explanation” can be formulated as follows:

An explanation is the process of describing one or more facts.

The third aspect of an explanation deals with its purpose. Ideally, the output of the explanation (i.e., the explanans) exposes patterns or statistics that were not evident before. For example, overlaying the gradients of an image classifier w.r.t. an input sample $\frac{\partial\mathcal{L}(f(\mathbf{x}),y)}{\partial\mathbf{x}}$ can locate areas that are more sensitive to changes in the input (e.g., by adding small levels of noise). Once more, resorting to textbook definitions (Table 2) we see that most of them mention “making something understandable, clear, comprehensible” as the goal of an “explanation”. In other words, the purpose of an explanation is to enable (human) understanding. Explanations are therefore bound to describe facts in a way that ultimately leads to understanding. At this point, we can think of what consumers of explanations (i.e., explainees) need to understand: on the one hand, there are characteristics of the fact being described (e.g., location of the magnitude and sign of a gradient w.r.t. an input sample) which may help one understand what the gradient is⁴⁴4Assuming that the explanee’s mental model is otherwise equipped with the necessary knowledge to understand this concept.. On the other hand, there are some characteristics of the described fact in relation to other high-level phenomena e.g., how a high gradient value relates to a latent relevant feature.

With this in mind, we arrive at a revised definition for “explanation”:

An explanation is the process of describing one or more facts, such that it facilitates the understanding of aspects related to said facts (by a human consumer).

The dependency between explanations and humans is explicit, as the action of understanding can be thought of as being unique to humans. Whenever explanations are consumed by other machines (or can be otherwise executed without humans in the loop) they are no longer serving as a vehicle to explain but rather as part of a verification system.

In order to constrain the many ways a description can be interpreted, a contract must be first introduced detailing the valid meanings that can be extracted out of that description. In other words, there should be an agreement on how to read the symbols of a description e.g., which colors on a heatmap mean high or low values. Such agreement anchors or assigns meaning to a primitive entity (in our case, an explanans).

Referring back to textbook definitions for “interpretation”, we see how most of them rely on the term “explanation”, and thus leading to an inexorable circular definition. The one remaining exception follows the philosophical origin of the word and already lays out the requirements of a contract by defining an “assignment of meaning”. In fact, the conveyance of meaning is common to all entries, in one way or another. We go along these lines to sketch the definition of interpretation in XAI as follows:

Interpretation is the assignment of meaning (to an explanation).

For ML, the assigned meaning refers to notions of the high-level task for which the explanans is provided as evidence. An interpretation is therefore bridging the gap between underspecified non-functional requirements of the original task and its representation in formal, low-level primitives (e.g., high Shapley values for pixels on a chicken’s beak indicate its correct detection and relation to the class “chicken”).

In order for an explanation to fulfill its goal (facilitate understanding), the terminology and symbols known to the explainee i.e., the consumer of the explanation, have to match those used by the explainer i.e., the proponent of the explanation, when assigning meaning to an explanation. In other words, the complexity (otherwise known as parsimony) of the interpretation should not be greater than the explainee’s capabilities to fathom its meaning (Ras et al., 2018; Sokol & Flach, 2020).

Sometimes, there are no fundamental reasons to prefer one interpretation over another (e.g., make red represent high values instead of blue or white) as long as one is agreed upon. There are scenarios in which interactivity allow for correction and negotiation of the meaning being assigned to explanations (e.g., arguing for the importance of using another color other than red for high values of a heatmap). Often, however, it is fundamental to consider the mental model of the explainee, as it expedites the process of understanding (i.e., it has an optimal degree of parsimony). This involves estimating the relevant mental constructs known to potential explainees beforehand, as exact mental models may not be available. In addition, mental constructs unknown to the explainee, but crucial for the interpretation, need to be properly introduced. In the worst case, ambiguities can lead to a misinterpretation of explanations⁵⁵5Prominent examples have even made it into mainstream media (Baraniuk, 2017).. A simple strategy to bridge the gap between the explainee’s mental model and the constructs of an interpretation is to rely on common conventions such as those coming from a natural universal language (King, 2005).

Note that the explainee’s mental model is assumed to be based on true facts; the consequences of engaging in the process of explanation and interpretation based on false premises can be catastrophic (Lombrozo, 2006). In consequence, matching an explainee’s mental model is desirable but not indispensable. Interpretations are valid as long as their statements cohere, and the inclusion of a particular explainee’s mental model should only accelerate the process of understanding, while leaving the essence of the interpretation itself unaltered.

Even if the methods to explain (or the agreements to interpret) vary, the process of understanding will always be supported by the same mechanism: description of facts followed by the assignment of meaning for the description itself. While the meaning being assigned can be contested (e.g., as part of the scientific process where one seeks to falsify a statement) the process of assignment cannot.

One of the main aims of equipping ML models with explanation capabilities is to contest previous beliefs w.r.t. a particular prediction by constraining an otherwise undetermined problem (Lombrozo, 2006). So far, we have already constrained two core definitions in XAI, in an effort to define a unified language that allows us to engage in scientific discussion. Said terminology still needs to fit into a more generic, procedural framework where novel and existing contributions can be placed and thus, compared.

Explanations and interpretations are ultimately aimed at answering questions arising from the underlying process (i.e., the ML model) that issued a prediction. These questions, in their more generic form, correspond to variants of what, how or why queries. Hence, our interest lies in defining a framework that addresses these questions when defining novel explanations and interpretations. In fact, we show that both terms are central for answering all three questions. We describe the scope of the aforementioned questions, and the role that explanations and interpretations both play when answering them.

The what defines the domains in which explanations operate. As defined in section ., we find that explanations, as processes, take in elements of a particular source domain, and output descriptions that exist in a target domain. This notion of “translation” between two domains has been recently promoted (Esser et al., 2020) although the terminology differs with the one proposed in this work. In mathematics, whenever a function is defined, it is first expressed in terms of the domain and co-domain where the function projects values into e.g., $f:\mathbb{R}^{d_{x}}\rightarrow\mathbb{R}^{d_{y}}$. Similarly, an explanation method should explicitly state what is being used as input and what is being produced as output. Ideally, the output of the explanation (explanans) should be defined in terms of low-level primitives that are as factually true as the input (e.g., intermediate features, support vectors, gradients). Note that, while inputs in the source domain are limited by the model being explained, the target domain depends on the explanation method, where options are virtually unlimited⁶⁶6Practical constraints arise from the cognitive limitations of human minds (e.g., the inability to imagine a tesseract)..

Defining how something is being done can be addressed from two levels of abstraction. First, at the system-level, there is the question of how the model arrives at a prediction. Second, there is the issue of how the explanans is produced. In other words, what are the details behind the process that transforms between domains defined by the what, and used by the explanation. The former is precisely the kind of inquiry that XAI methods try to answer and therefore, they cannot be included in our framework. Instead, this kind of how questions will be answered through methods that rely on it⁷⁷7In fact, all three questions–what, how, why–have been proposed as the basis for identifying questions that are answerable through explanations (Miller, 2019).. The latter on the other hand, deals with the context of explanations and their answer is essentially reduced to the explanation method itself. Simply put, the answer of how the explanandum is mapped to an explanans is defined by the explanation method.

As most XAI literature is devoted to the development of explanation methods (e.g., computation of relevance values (Bach et al., 2015; Vaswani et al., 2017), mapping of high-level conceptual constructs in intermediate activations (Bau et al., 2017; Kim et al., 2018)), the how is almost always thoroughly defined. This has already nurtured insightful debates on whether linear approximations of the original model (Lundberg & Lee, 2017) or even other black-boxes (Al-Shedivat et al., 2020) can be considered appropriate explanation methods (Rudin, 2019; Slack et al., 2020).

Together, the what and the how already constrain the scope in which explanations are valid. The one remaining aspect is the context in which an explanation is interpreted. An explanans is, by itself, only relevant within the target domain defined by the explanation. It is the interpretation (of the explanans) which will map the low-level representations into a high-level domain where expectations regarding non-functional requirements can be validated or contested. Consider the following interpretation: “the result of an argmax operation on the logits of a neural network corresponds to the predicted class”. The high-level action of predicting a class has been mapped to an argmax operation over a vector. Based on this interpretation of the argmax operation, users of neural networks can either accept this notion to gather results, or find shortcomings and propose alternative ways of fulfilling the requirement of a model to predict classes (e.g., through the refinement of the prediction through a hierarchical exclusion graph (Deng et al., 2014)). The same principle can be applied to explanans and non-functional requirements.

Asking why something happens, inescapably relates back to causal effects. While these kind of relationships are among the most useful to discover (as it allows for more control over the effect by adjusting the cause), other non-causal relationships remain valuable in the toolset of explainability. Finding out that a model is unfair, without knowing what the cause of it is, can already be helpful in high-stake scenarios (e.g., by preventing its use). We say that the why relates to the nature of an interpretation. In short, if the interpretation bares a causal meaning, then the why is being defined by the causal link. If the meaning is limited to a correlation, the why is left out of the scope of that particular interpretation. Note that, if the explanation method is already based on causal theory (Lopez-Paz et al., 2017; Chang et al., 2019), the assigned meaning (i.e., the link between the explanans and the high-level (non-)functional requirement) will be more direct and therefore, more likely to withstand scientific scrutiny. The why is therefore not mandatory in explanations generated by XAI methods. In any case, the explanation’s context can and should be defined, be it causal or based solely on correlations. Proponents of XAI methods are responsible for clearly stating the context in which their explanations can be interpreted.

The explanandum of both LIME and SHAP comprises the prediction of the target model, as well as an input sample⁸⁸8Most local explanation methods rely on individual samples from the model’s input space as input for the explanation too. Non-local methods like TCAV (Kim et al., 2018) or S2SNet (Palacio et al., 2018) rely on a group of samples or even a representative sample of the input space.. The target model is treated as a black-box, and typically deals with low-dimensional input data. Meanwhile, MDNet’s textual and visual explanations are derived from an input sample, and from high-dimensional latent variables found within the target model. LIME and SHAP represent the target model’s low-level internal processes mainly through counterfactual analysis, while methods exposed to the model’s internals can examine the flow, translation and attribution of information as it traverses the model, serving as a much more direct evidence of a model’s decision process. In fact, it has been shown that limiting access to the target model’s internal representations makes the creation of adversarial attacks possible, compromising the reliability of the explanation (Slack et al., 2020). In other words, it is not enough to rely only on the input domain (of the target model) as the explanandum.

The outputs of both LIME and SHAP consist of the weight values from linear surrogate models that are mapped to their corresponding input region (visualized as heatmaps). Instead, MDNet’s explanans is composed of a sequence of one-hot encoded words from the language model, where each word is paired with a $14\times 14$ weight matrix that is spatially mapped to the input image (also visualized as a heatmap). The combination of generated text supported by word-wise attribution maps provides a traceable structure from the model’s input to the weight matrix. While LIME and SHAP provide explanans that share some of the output space with MDNet’s, the relation they bare with any internal representation of the target model is not as traceable.

LIME and SHAP both follow similar explanation strategies that are characterized by continuously perturbing and evaluating an input sample via the target model; results are subsequently approximated through a surrogate linear model. SHAP poses additional constrains to the surrogate’s optimization and slightly differs in its sampling scheme. MDNet can be described as a two stage process: an image feature extraction followed by a concurrent generation of a sensitivity map (originally called “explanation”) and textual diagnosis. Both the explanation and classification components of MDNet are trained simultaneously in an end-to-end fashion. The language model for text generation is trained using diagnosis texts as the supervisory signal, fulfilling one of its functional requirements. The word-wise attention module learns a set of sensitivity weights with additional constrains on diagnostic labels serving as indirect supervision.

Given all structural and low-level components of the explanations under scrutiny, as described by the what and the how, we now turn to the interpretation of such primitives. The first observation is that none of the aforementioned methods provide interpretations of causal nature. LIME and SHAP cannot provide interpretations grounded in causality due to issues related to off-manifold sampling (Frye et al., 2020) and the non-excludable inaccuracy when relying on proxy models (Rudin, 2019). Nevertheless, there are non-causal interpretations worth examining.

The weight values derived from LIME’s proxy models are interpreted as “influence values”. These values convey the relevance of individual input features w.r.t. the prediction of the target model. A small caveat to this interpretation is its limited validity, which applies only to the local neighbourhood around the original input sample. SHAP’s explanans, despite baring evident similarities with LIME’s, allows the generation of additional global explanans through an accumulation of statistics from multiple local explanations. Furthermore, the notion of feature attribution (i.e., how much the value of each input feature has influenced the model’s prediction) inherits the properties of Shapely values. In this case, values are interpreted as payouts to individual features, reflecting how much they contributed to the model’s prediction, relative to the remaining input features. MDNet’s visual explanations are interpreted as the model’s attention w.r.t. a region within the input image while the textual-diagnosis generates a word. In this case, we see how MDNet defines explanans as part of the model’s output and not as a separate process, tying predictions and explanans together. The correspondence between feature attribution on the image domain and the sensitivity w.r.t. generated text comes from additional annotations available for the text. The assumption is that latent representations of MDNet align features of image regions and text in a way that is meaningful to humans. The link between the intuitive notion of “attention” and its implementation in one of MDNet’s modules, has been recently contested (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019), undermining any causal relation drawn from it.

In the process of defining the proposed framework, we conducted an extensive review of XAI literature. This allowed us to identify current practices in the XAI community, some of which we summarize here in terms of our framework.

• •

  Most contributions focus on the development and use of explanation methods while neglecting the role of the interpretation (with notable exceptions like  (Rudin, 2019; Montavon et al., 2018)).

• •

  Explanations are often defined without explicit definitions of the domain and co-domain (i.e., realm of the explanans and explanandum).

• •

  More recently, explainable models are trying to include richer objectives through auxiliary tasks: this strategy addresses both the underspecification of the task (conveys some of the sought after non-functional requirements) and expresses some of the invariances that are expected of the task.

• •

  In domains where AI takes over or assists human-professional workers (e.g., for medical applications), interest often shifts from the prediction to the explanation such that human experts learn and gain new insights about the task.

Finally, we identify three aspects that future research in XAI should focus on in order to expedite the advancement of the current state-of-the-art.

• •

  Before working on the specifics of an explanation, explicitly define the non-functional requirements that a task should fulfill, and that the explanation itself will be probing for. This can be achieved by defining better hypotheses e.g., one that can be falsified (Leavitt & Morcos, 2020).

• •

  Proponents of XAI methods should be careful when addressing the complete scope of our proposed framework. In particular, a clear interpretation should be provided.

• •

  Metrics for XAI methods should operate within the same level of abstraction w.r.t. the framework i.e., compare explanans to explanans, explanandum to explanandum, interpretation to interpretation, etc.