Explainable Deep Reinforcement Learning: State of the Art and Challenges

M. T. Ribeiro, S. Singh and C. Guestrin in (LIME, ) propose LIME for explaining the predictions of any classifier in an interpretable and faithful manner. LIME learns an interpretable model locally around the prediction, searching for instances in the vicinity of the original instance through perturbations of uniformly sampled features in subsets of $M$ (i.e., non-zero elements). Emphasis is given to the qualitative understanding of correlations between input features and model responses, aiming to guarantee local fidelity, explaining how the model behaves in the vicinity of the instance being predicted, using a proximity measure. Thus, while LIME addresses the model inspection problem assuming binary vectors for instances’ interpretable representations, it contributes to the outcome explanation problem solution, as it provides the importance of features in a local only context.

Building on explanations of individual predictions, authors in (LIME, ) propose the SP-LIME method by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a sub-modular optimization problem. SP-LIME provides a global perspective and contributes to solving the model explanation problem.

Experiments performed with simulated users regard (a) the faithfulness of explanations to the model, given a set of gold features that explanations must recover; (b) the trust on predictions, given untrustworthy features in predictions; (c) the trust on the model, given a set of trustworthy examples covering cases with varying subsets of important features. Evaluations with human subjects regard the accuracy of choosing the best classifier, and improving the classifier by eliminating features which are untrustworthy (measuring classification accuracy after rounds of interaction).

LIME provides a generic (model-agnostic) method that contributes to local and global (model) interpretability. In the context of DRL, it can be used for explaining any of the models (e.g., to explain local decisions for selecting an action given an interpretable representation of states’ features), although not being designed for that particular task. Functions $f(\cdot)$ provide binary vectors for instances’ representations, while $c_{l}$ (for outcome explanation ) can theoretically be any interpretable model, e.g., a linear model, a decision tree, etc. Surface representations of instances’ interpretable representations (for model inspection, via $R$), as well as explanation logic $\epsilon_{l}$ for local explainability and $\epsilon_{g}$ for model explainability, are provided using visual means.

Extensions of LIME include Anchor (ANCHOR, ) that uses decision rules as local interpretable classifier $c_{l}$, and LORE (LORE, ) that learns a local interpretable predictor $c_{l}$ on a synthetic neighborhood using a genetic algorithm.

The Shapley Additive Explanations (SHAP) approach aims to provide a unified approach to interpreting model predictions, showing how interpretation methods are related, and when one method is preferable over another. S. M. Lundberg and L. Su-In in (SHAP, ) identify the class of additive feature attribution methods with a linear function of binary variables as an explanation model: $g(z)=\phi_{0}+\sum_{i=1}^{SIF}\phi_{i}z_{i}$, where $z\in\{0,1\}^{SIF}$, $SIF$ the number of simplified input features and $\phi_{i}\in R$.

There is a family of methods - including LIME, DeepLIFT (DEEPLIFT, ), Layer-Wise Relevance Propagation (LayerWiseRelPropagation, ) and classic Shapley value estimation methods- that are additive feature attribution, and authors in (SHAP, ) show to unify six of the existing methods. They provide theoretical results on the existence of a unique solution in this class of methods with a set of desirable properties, that several recent methods in the class lack: Local accuracy, requiring the explanation model to at least match the output of the original model; missingness, representing features presence so as to locate the features with no prediction impact on a local decision; and consistency, regarding the relation of input’s contribution and input’s attribution.

Specifically, considering a simplified input $x^{\prime}$ mapping to the original input $x$ through the mapping function $h_{x}$: $x=h_{x}(x^{\prime})$, and input $z^{\prime}\approx x^{\prime}$, with $S$ to be the set of non-zero indexes in $z^{\prime}$ ($z_{S}$ has missing values for features not in $S$) s.t. $h_{x}(z^{\prime})=z_{S}$, SHAP values are the Shapley values of a conditional expectation function of the original model $E[pred(z)|z_{S}]$, given the model prediction $pred(z)$. SHAP focuses on explaining a prediction $pred(x)$, given a model $pred$ with input $x$ and a simplified input $x^{\prime}$. Such local methods ensure that $g(z^{\prime})\approx pred(h_{x}(z^{\prime}))$, whenever $z^{\prime}\approx x^{\prime}$, where $g$ is the local interpretable model $c_{l}$ (for outcome explanation). SHAP, similarly to LIME, provides an interpretable model as a linear function of binary vectors for instances’ representations provided by $f(\cdot)$. Surface representations $R$ of instances’ interpretable representations for model inspection, as well as explanation logic $\epsilon_{l}$ for local explainability, are provided using visual means.

Based on insights from unifying methods, authors in (SHAP, ) present methods that show improved computational performance and/or better consistency with human intuition than previous approaches. Evaluation of Kernel SHAP vs. LIME and Shapley sampling values show improvement in terms of learning accurate estimates with fewer evaluations of the original model (sample efficiency). User studies in a limited setting show that SHAP values prove more consistent with human intuition than alternative feature importance allocations represented by DeepLIFT and LIME, which fail to meet the desirable properties identified. Finally, authors use MNIST digit image classification using a convolutional network to compare SHAP with DeepLIFT and LIME, showing that SHAP is able to explain differences in classifying a digit to a class rather than to a near-class, measuring the probability of classifying instances to classes and changes in log-odds when masking over images.

A concrete example of employing SHAP (actually, a specific instance of LIME, named Kernel SHAP) in the context of DRL, is provided in reference (XRL4TrafficSignals, ). This article uses Policy Gradient to implement a traffic signal control agent on a signalized roundabout. Kernel SHAP estimates the effect of the road detectors state on the agent selected phases, interpreting the decision taken, in relation to the traffic volumes and the lanes occupancy.

Other methods, such as for instance Layer-wise Relevance Propagation (LRP) (LayerWiseRelPropagation, ) and Neighborhood Component Analysis (NCA) (NCA, ) can be used for addressing the model inspection problem for any of the constituent DRL models, also providing (a) importance of features in $M$, and (b) reduction of predictors’ dimensionality, which can facilitate data visualization and fast classification. Such methods are thoroughly reviewed in (Survey, ).

Following this short presentation of model inspection general methods, we proceed to review XDRL methods addressing any of the model, outcome explanation, or the interpretable box design problems.

With the goal of increasing people’s familiarity with agent’s capabilities and limitations, D. Amir and O. Amir in (Highlights, ) propose the HIGHLIGHTS and the HIGHLIGHTS-DIV algorithms that choose trajectories highlighting important and distinct aspects of agent behaviour. In doing so, this approach provides an overview of an agent’s policy, rather than explaining specific responses.

Specifically, the HIGHLIGHTS online algorithm constructs summaries of agent’s behaviour by selecting (sub-) trajectories with important states. A state is considered important if taking a “wrong” action in that state (i.e., an action that is not prescribed by the policy) can lead to significant decrease in future rewards. To provide context to the user, for each important state the algorithms extracts a sub-trajectory including neighboring states (before and after the important states) and actions. In order to highlight behaviour in various sub-spaces in the state space, HIGHLIGHTS-DIV avoids including in the summary (sub-) trajectories with very similar states, if these are equally important.

HIGHLIGHTS has been evaluated by playing the Pacman game, based on the observed behaviour rather than scores. Q-values in this setting are defined as a weighted function of state feature values. Given different agents trained for different number of episodes, human participants were asked to select the best agent based on the summaries provided, while subjective opinions about the helpfulness of summaries were elicited, given summaries of the same agent. The surface representation of the explanation in (Highlights, ) is provided by means of video clips, but in general, other means and information modalities may be used (e.g., as in (ContrastiveExplanations, ) that is presented in a subsequent section). This evaluation resulted in observations regarding participants confidence in choosing agents based on HIGHLIGHTS summaries. Experiments show that short summaries may not enable participants to correlate their confidence with their ability to assess agent’s abilities. However, with statistical significance and for very well trained agents, participants preferred summaries generated by HIGHLIGHTS over baselines, and summaries generated by HIGHLIGHTS-DIV over HIGHLIGHTS.

Overall, this approach addresses5 the policy model explanation problem, demonstrating agent behaviour. It is important to point out that this approach by-passes the construction of an interpretable model $c$ for policies, but it gathers explanations’ content by selecting examples of agent’s behaviour in important states. The representation of importance is realized via the function $f(\cdot)$ which has access to agent’s policy and Q-values.

T. Huber et al. in (LocalAndGlobal, ) use a version of HIGHLIGHT-DIV in a combination of global and local explanations for DRL agents. Specifically, they use a version of the Layer-wise Relevance Propagation (LRP) (LayerWiseRelPropagation, ) method, i.e. LRP with the argmax rule (LRP-argmax, ), to augment trajectory summaries with saliency maps. This local approach is presented in detail in section 4.4.11. LRP has been used to show what information is important to the agent towards a specific decision: The approach presented in (LocalAndGlobal, ) addresses the policy and the responses explanation problems by combining methods, contributing to our understanding of the role of saliency maps in the context of explaining agents’ behaviour: While saliency maps have been shown to improve classification decisions in images (albeit with 60% of correctness in (Alqaraawi4, )), their role in RL is not that clear (e.g. as shown in (TransparencyandExplanationInDRL, )). However, they may provide significant positive effects when combined with other methods (as in (Erwig23, )). Empirical evaluation of the proposed combined method aims to investigate (a) the mental model of humans about the DRL agent (via a retrospection task), (b) humans’ ability to assess agents performance (via an agent comparison task) and (c) participants satisfaction with respect to the explanations presented (via answering explanation satisfaction questions). The results of this study reinforce prior findings for HIGHLIGHTS-DIV. Overall, in this study, the choice of states shown to humans was more important than the inclusion of local explanations in the form of saliency maps. For saliency maps, the results reported are mixed: There were no significant differences between the saliency and nonsaliency conditions in the study. Showing saliency maps as part of a video is more challenging for humans, in comparison to showing them in a still image. As authors point out “[saliency maps] in RL [add a] layer of complexity as interpretation also requires making inferences regarding how the highlighted regions affect the agent’s long-term, sequential decision-making policy”. However, in combination with behaviour explanations, saliency maps put the important features in context, while they are not preferable when humans get highlights of agents’ behaviour.

Extending the HIGHLIGHTS algorithms, P. Sequeira and M. Gervasio in (InterestingnessElements, ) propose using not only state importance, but various interestingness elements so as to highlight important agent’s behavioural aspects, increasing humans’ understanding of agent’s capabilities and limitations. Such elements are (in)frequent situations, (un)certain executions, observation minima and maxima, and most likely sequences to maxima, each serving a specific explanation provision purpose. To compute these elements, the proposed framework collects data from agent’s interaction with the environment, such as the number of times the agent has specific observations, estimated transition probabilities, as well as estimated Q-values and V-values. Although there are elements (e.g., frequency) that may provide a good understanding of agent’s behaviour characteristics, overall, results show that no single summarization technique — and hence no single interestingness element — provides a complete understanding of every agent in all possible situations of a task. Ultimately, different combinations may be required for agents with distinct capabilities and levels of performance.

Motivated by findings indicating that contrastive explanations offer intuitive motivations for humans to understand why one performs an action instead of another (Miller, ), J. van der Waa et al. in (ContrastiveExplanations, ) propose a method for providing contrasting descriptions of rollouts generated by an RL agent’s policy and a policy that is driven by humans’ (non-experts in RL) queries.

This proposal (a) defines a translation of states and actions in a set of descriptive states’ classes and action outcomes, by training binary classifiers, similarly to (AutonomousPolicyExplanation, ), and (b) proposes a method for obtaining a foil policy $\pi_{f}$ based on the foil in the user’s query, suggesting potential alternative actions in a particular state $s$ and consecutive state-action pairs. This foil policy, based on an update of the $Q$ function to reflect user’s preferences (and an informed update of the reward function) is contrasted to the fact policy $\pi_{t}$ learned by the agent. The explanations are based on generating rollouts of specific length simulating the effects of $\pi_{t}$ and of $\pi_{f}$, based on the environment’s transition function and starting on a state before the state specified in the user’s query. State-action pairs in rollouts are transformed in descriptions which are then used for generating the trajectory contrastive descriptions by instantiating natural language templates.

Authors in (AutonomousPolicyExplanation, ) report on a case study using a simple RL setting with a very restricted objective, addressing either the next action of the agent policy or the entire policy: They show that human participants prefer explanations using the policy, as in most cases the next action is evident to the user.

This approach addresses the policy model explanation problem, by contrasting rollouts of the agent’s policy to foil policy rollouts. The interpretable model $c$ provides qualitative descriptions of rollouts by means of distinct state and action classes. The interpretation process accesses the MDP, which may be fitted by the learning process, the state-action Q-values learned, and the rewards received, via the function $f(\cdot)$. Finally, the explanation logic $\epsilon$ uses natural language templates that are instantiated by qualitative (potentially, contrasting) descriptions of rollouts.

Motivated towards helping end-users to build a mental model of agent’s policies and capabilities, S. H. Huang et al. in (CriticalStates, ) propose approaches for computing critical states in a policy-agnostic way, requiring only access to action-value function $Q$. This proposal is driven by the insight that for many tasks the essence of the policy is captured by agent’s reactions in a few critical states.

A critical state $s$ is one in which it is very important to take a certain action: As defined in (CriticalStates, ), $Q^{\pi}(s,a)$ varies greatly across different actions $a$, i.e., $Q^{\pi}(s,a)$ is very high for some actions, but for most of them, this value is mediocre or low. Given agent’s policy, critical states are exposed to end-users who can spot false-positive or false-negative critical states (i.e., states that are considered as critical by the agent, but not by the humans and vice versa). Inspecting the critical states, the end-users may refine their trust to the agent, or gain proper knowledge on the situations where they have to take control. For instance, both false-negative and false-positive critical states indicate that the robot has failed to learn something fundamental about the task. Similarly, if the policy identifies a true-positive critical state but is mistaken about which action is correct in that state, then the end-user may not trust the policy.

This approach does not provide an explicit interpretable model but aims to provide a coherent view of agent’s capabilities and limitations so as to establish trust to the agent. Critical states are determined by accessing $Q$-values via the function $f(\cdot)$. The interaction with human experts concerns (a) spotting false-positive, false-negative states, or incorrect-actions in critical states and assessing whether to deploy the agent, (b) deploying the agent which operates with the user in the loop, who she is able to take control whenever needed. A critical issue is whether observing a set of critical states leads participants to develop appropriate trust in high-entropy policies that generated those critical states, using the Soft Actor Critic algorithm (SAC). Also, while we may consider that this approach addresses the policy model explanation problem, it may be hard for the human to generalize and consider how the agent reacts in states that have not been presented as critical. Also, while the surface representation of states-action pairs is shown to be depicted using visual means, it is harder to do so in settings where the dimensionality of the state/action space is large.

The main observation driving the work of T. Zahavy, N.B. Zrihem and S. Mannor in (GrayingBlackBox, ) is that Deep Q-networks (DQN) is learning an internal hierarchical model of the domain without explicitly being trained to. Therefore, authors propose a methodology and tools to analyze DQN, aiming to identify spatio-temporal abstractions directly from the learned representation in two different ways: First, manually clustering the state space using hand crafted features, and second, computing the Semi-Aggregated Markov Decision Process (SAMDP), an approximation of the true MDP learned from data, by means of spatio-temporal abstractions. Visualization tools, including t-SNE that exploits directly the collected neural activations of states (a model inspection task), saliency maps, as well as analysis tools for understanding the common attributes of states’ clusters, and for analyzing dynamics between clusters, aim to provide facilities towards understanding, debugging and interpreting the policy model.

Specifically, authors in (GrayingBlackBox, ) explain the five stages towards building an SAMDP model aggregating states and skills, thus allowing analysis with spatio-temporal abstractions: (0) Feature selection; (1) State aggregation, mapping the MDP feature space to the abstracted MDP state space enforcing temporal coherency among trajectories; (2) Identification of skills (a.k.a sub-trajectories) from the data; (3) Inference of skill length, the SAMDP reward and the SAMDP transition probabilities; and finally, (4) selecting the best among the SAMDP candidates, according to evaluation criteria proposed.

Overall, this approach proposes a methodology and abstraction methods addressing the policy model explanation problem for DRL methods, where the policy is learned by the DQN method. The interpretable model $c_{g}$ is an SAMDP that is devised by consulting manually crafted state features, as well as rewards and neural activations of states provided by $f(\cdot)$. The explanation logic $\epsilon_{g}$ is implemented by t-SNE maps with representative states per cluster, and appropriate t-SNE enabled SAMDP visualizations. In conjunction to that, authors demonstrate how to do (policy) model inspection by processing the network’s neural activity using visualizations enabled by t-SNE.

R. M. Annasamy and K. Sycara in (betterInterpretability, ) propose an interpretable NN architecture for $Q$-learning, which provides explanations for visual agents. The aim is to provide an understanding of the policy model, visualizing clusters of state representations in attention maps, and using saliency maps.

The proposed model learns a latent representation of states that captures important visual aspects of input images, and an association between representations of states and representations of keys in a key-value store using an attention mechanism. Each key represents an action and an associated $Q$-value, among $N$ such values: Thus, the network learns to associate states to “representatives” of (action,$Q$-value) pairs, which serve as cluster ”exemplars”. The attention weights over keys and their corresponding value pairs are used to calculate $Q$-values. Different types of losses are being linearly combined to train the network: Bellman error, distributive Bellman error, reconstruction error and diversity (over keys) error. Uncertainty of the attention weights is being used to drive exploration during training, approximating an upper confidence on the $Q$-values. Through key inversion (which not a process driven by a certain input) authors attempt to find important aspects of the input space that influence the choice of a particular action-return pair.

As concluded in (betterInterpretability, ), with a directed exploration strategy, the proposed model can reach training rewards comparable to the state-of-the-art deep $Q$-learning models. Specifically, the best agreement between the actions selected using the distributions in the image space and latent space is rather low, but the explanations provide useful insight into the kinds of features extracted by convolutional layers: This possibly suggests, according to the authors, that reconstructions rely heavily on generalizability to unseen keys. However, results suggest that the features extracted by the NN are shallow, and the agent does not model interactions between objects.

Concluding, this approach addresses the interpretable box design problem for deep $Q$-learning methods, focusing on providing global policy explanations for visual agents. The interpretable model $c_{g}$ is an attention model associating state embeddings to key-value embeddings. The explanation logic $\epsilon_{g}$ provides keys - state embeddings clusters’ visualizations using t-SNE for any specific $Q$-value, as well as images reconstructed via key-value (i.e., action, $Q$-value) pair inversion.

Off policy evaluation (OPE) estimates the value of a policy using data collected under another policy. Aiming to enable human experts to assess and analyze the validity of off-policy evaluation estimates, O. Gottesman et al., in (InterpretableOPE, ) propose a hybrid human-AI system that highlights observations in the data whose removal will have a large effect on the OPE estimate. Their proposal aims (a) to address the needs of real-world applications where the data itself might never be enough; and (b) to involve domain experts in the integration of decision support tools, incorporating their expertise into the evaluation process. In so doing, they formulate a set of rules for choosing which observations to present to domain experts for validation.

Specifically, in this article authors propose a framework for using influence functions to interpret OPE, and discuss the types of questions which can be shared with domain experts to use their expertise in debugging OPE: Influence functions show the impact that the removal of a transition has on state-action values or on the value of the evaluation policy. Based on this framework, they develop computationally efficient algorithms to compute the exact influence functions for several importance sampling estimators, as well as two broad function classes for Fitted Q-Evaluation (FQE): kernel-based functions and linear functions.

The benefits of influence analysis are demonstrated on a cancer simulator, with the objective to identify limitations in the evaluation process and make evaluation more robust. Furthermore, authors present results of analysis together with practicing clinicians of OPE for management of acute hypotension from a real intensive care unit (ICU) dataset.

Overall, this approach proposes a framework that addresses off-policy evaluation independently from any reinforcement learning method. It is closer to the policy explanation problem, since the influence analysis proposed aims to interpret a policy evaluation method. The method can be applied to parametric and non-parametric fitted Q-evaluation models, regarding continue or discrete states and actions, although the presentation and evaluation considers discrete cases. The method focuses on the influence of specific state transitions to the value of the policy, and thus, it can be considered to be a method that reveals the contributions of local decisions to the trajectories generated by a policy model. The model exploits trajectories generated by the evaluation policy, while there is not any specific proposal for the explanation logic: Different examples, use different visualizations of transitions’ influences.

A.Mott et al. in (AttAugmentedAgents, ) propose a spatial attention mechanism to visual information in an RL setting. The model enables agents to actively select important, task-relevant information from visual inputs, by sequentially querying and receiving compressed, query-dependent summaries. Actually, the model generates attention maps which can uncover the underlying decision process. Exploiting the attention maps one can understand how the system solves a task, identifying the type of entities that the agent attends (“what”) and the spatial locations where the agent focuses attention (“where”).

Succinctly, the proposed model works as follows: Observations pass through a recurrent vision core network, producing a “keys” and a “values” tensor, to both of which a spatial basis tensor (encoding information about spatial positions) is concatenated. A query network produces a set of query vectors taking as input the state of an LSTM recurrent network (policy network) from the previous time-step. The softmax of the pixel-wise inner product between each query vector and each location in the keys tensor produces an attention map for the query. The attention map is broadcast along the channel dimension, and it is point-wise multiplied with the values tensor. The result is then summed across space to produce an answer vector. This answer is sent to the LSTM to produce the next LSTM state.

The performance of the proposed method is evaluated across a broad range of ATARI levels. Attention maps are used to visualize which parts of the input are attended. Authors conjecture that the top-down nature of the attention (i.e., the fact that the attention map is produced by taking as input the LSTM states) provides a large performance gain compared to equivalent, bottom-up attention-based mechanisms. Comparison of the attention maps to alternate methods for visualizing saliency shows that they allow more comprehensive analysis of the information the agent is using to inform its policy. The agent is able to make use of a combination of “what” and “where” queries to select both regions and entities within the input, depending on the task. Experiments show that the agents are able to learn to focus on key features of the inputs, look ahead along short trajectories, and place tripwires to trigger certain behaviors.

This approach proposes a model addressing the interpretable box design problem for DRL methods applied in visual inputs, where the policy is modeled by an LSTM network. The method focuses on providing global policy explanations. The model exploits LSTM states to produce queries, as well as keys and values tensors produced by the visual input: Thus, we could say that the query network and the vision core serve as the $f(\cdot)$ to feed the interpretation component. The interpretable model $c_{g}$ is a soft attention model and the explanation logic $\epsilon_{g}$ is implemented by visualizing the attention maps, showing the original input frames and super-imposing the attention map for each attention head. In addition, saliency maps, visualizing the relative dominance of “what” and “where” parts of the query are being used, to inspect where the agent focuses attention.

Decision trees that represent policies span the state space and may be larger than required, even if they do express the action value function quite accurately. A. M. Roth et al. in (ConservativeQImprovement, ) aim to provide an answer to the question “what should be the size of such a decision tree” to make it accurate and interpretable? This work is based on the hypothesis that the size should be that, which if increased further it does not increase significantly the estimated discounted future reward of the overall policy.

In doing so, Roth et al. in (ConservativeQImprovement, ) propose the Conservative Q Improvement (CQI) RL algorithm. CQI learns a policy in the form of a decision tree, in any domain with discrete actions and multidimensional state space. In contrast to other RL methods (e.g., (Pyeatt, )) that learn a decision tree policy representation, this approach uses a lookahead approach that predicts which split will produce the largest increase in reward. The method is strictly additive: Initialized with a single leaf node, over time, it creates branches and leaf nodes by replacing existing leaf nodes with a branch node and two child leaf nodes. A branch node represents abstract states and indicates actions to be taken using the $Q$ function. A leaf node is split (and turned into a branch node) only when the expected discounted future reward of the new policy would increase above a dynamic threshold, moderated by the node visit frequency and the dynamic split threshold.

CQI is evaluated in the RobotNav domain, where a robot must navigate in a 2D environment to reach a goal location, avoiding obstacles. It is shown that the method is able to produce substantially smaller trees compared to the method of (Pyeatt, ), with smaller variance and of greater reward. No explanation logic $\epsilon$ is proposed in reference (ConservativeQImprovement, ), and this article does not evaluate the method in terms of interpretability or explainability.

CQI follows the interpretable box design paradigm: It aims to learn an interpretable policy model $c_{g}$ in a direct way towards increasing explainability.

T. Shu et al. in (HierarchicalinterpretableRL, ) propose a framework for efficient multi-task RL, training agents to employ hierarchical policies: These policies support an agent to decide when to use a previously learned policy and when to learn a new skill. Each learned task has a corresponding human language description, thus represented in an interpretable way. Through the hierarchical model proposed, one may i) accumulate tasks progressively from a terminal policy to a top-level policy, and ii) unfold the global policy from top-level to basic actions, using human language descriptions. This approach not only learns the language grounding for visual knowledge and policies, but is also trained to utter human instructions as an explicit explanation of its decisions to humans.

More specifically, training progresses at stages, increasing the set of tasks from $\mathcal{G}_{0}$ to $\mathcal{G}_{k}$ at stage $k$, and the learned policies from $\pi_{0}$ for $\mathcal{G}_{0}$, to $\pi_{k}$ for $\mathcal{G}_{k}$. At stage $k$, the policy $\pi_{k}$ is defined by a hierarchical policy that consists of four sub-policies: a base policy for executing previously learned tasks (the policy at stage $k-1$), an instruction policy that manages communication between the global policy and the base policy (mapping a task $g\in\mathcal{G}_{k}$ to a base task $g^{\prime}\in\mathcal{G}_{k-1}$), an augmented flat policy which allows the global policy to directly execute actions instead of using a base task, and a switch policy that decides whether the global policy will primarily rely on the base policy or the augmented flat policy. A stochastic temporal grammar (STG) model is trained on the sequence of policy selections (previously learned skill or new skill) and focuses on priorities among tasks: It interacts with the hierarchical policy through modified switch policy and instruction policy. Learning at each stage is a 2-phase curriculum learning task: Policies in each phase are learned with an advantage actor-critic (A2C) off-policy method, assuming that the terminal policy $\pi_{0}$ has been trained, with base skill acquisition (learning to use previously learned skills) and novel skill acquisition phases (learning to rely on the augmented flat policy for executing novel tasks).

The approach is evaluated in a two room environment in Minecraft with 24 tasks in total. Experiments aim mainly to show learning efficiency and generalization abilities. Hierarchical plans of several tasks generated by global policies learned by the full model are visualized. However, these visualizations are mainly used to show agent’s learning efficiency, rather than the provision of explanations.

Concluding, this approach follows the interpretable model design paradigm, training interpretable models that are hierarchical plans, where skills and tasks are described in natural language based on human instructions.

The Logic-based Value Iteration Networks (LVIN) proposed by T. Leech in (LVIN, ) combines imitation learning and learns to represent problems as compact finite state automata (FSA) with human-interpretable logic states. The aim is to facilitate understanding and manipulation of the learned model, towards adding trust and flexibility to robotics applications trained with LVIN.

LVIN produces a policy which describes desired actions in terms of transitions between human-interpretable logic states. In doing so, the planning problem posed by the decision layer is decomposed into two pieces, a high level finite state automaton (FSA) that corresponds to high level sub-goals the robot follows (described in logic) and a low level Markov Decision Process (MDP) describing the motion of the robot in the physical environment. The FSA learns the rules governing the robot’s behavior in terms of interpretable and manipulable logic states, and transitions, called propositions. LVIN comprises separate Value Iteration Networks for each FSA state, where learning the state transitions associated with the high level FSA involves encoding the order of propositions to trigger given the current FSA state, and learning the $P(s^{\prime}|s,a)$, $s\in S$ and $a\in A$, associated with the low level MDP. LVIN learns the entire model via an iterative update process, similar to Value Iteration Networks, where the update procedure is encoded in a deep NN.

Evaluation aims to show the success rate (i.e., the percentage of correctly completed tasks) of LVIN compared to a CNN trained with a map as input and the desired action as output. The method has been evaluated in a lunchbox packing and in a cabinet checking problem, with no emphasis on evaluating interpretability or explainability.

LVIN addresses the model explanation problem for discrete actions and in a 2D space. The interpretable global model $c_{g}$ is the high level FSA. In this case, the function $f(\cdot)$ provides the “raw” model (deep NN) parameters, i.e., making no interpretation. Since the high level FSA and the low level MDP are trained ”seamlessly” taking as input the same expert demonstrations, learning the interpretable model can be considered as a mimicking process. LVIN does not propose any specific explanation logic for realizing the function $\epsilon_{g}$. Abstract policies in terms of logical states and transitions explain low level policies, assuming that these abstractions are interpretable by humans. As authors point out “the current LVIN framework … is still somewhat limited by the fact that it requires a hard coded logic oracle for training”.

G. Liu et al. in (interpretableDRLwithLMUTs, ) introduce Linear Model U-trees (LMUTs) to approximate NN predictions for DRL agents. An LMUT is learned by an on-line algorithm that is well-suited for an active play setting, where the mimic learner observes an ongoing interaction between the DRL and the environment. The article shows how the interpretable tree structure of an LMUT facilitates analyzing feature influence, extracting rules, and highlighting the super-pixels in image inputs.

U-tree (U-tree, ) (CUT, ) is a classic online RL method which represents a $Q$ function using a tree structure. In addition to that, continuous U-tree (CUT) (CUT, ) dynamically generate a tree-based discretization of the input signal and estimates state transition probabilities by retaining transitions in every leaf node. To strengthen the generalization ability of these representations, as well as their learning efficiency, Linear Model U-Trees (LMUT) (interpretableDRLwithLMUTs, ) add a linear model to each leaf node. This can be trained either in a batch mode or in an active play setting. LMUT applies Stochastic Gradient Descent to update the linear models, given some memory of recent input data stored on each leaf node. In the active play setting the method exploits transitions in the form of $(state,action,Q-value,next-state,reward)$, where the $action$ is decided by the DRL model, together with the “soft” $Q-value$, while the $reward$ is provided by the environment.

LMUT builds an MDP from the interaction data between the environment and a deep model. Compared to a linear Q-function approximator, a Linear Model U-Tree (corresponding to an action) defines an ensemble of linear Q-function models, one per leaf node and one for each state partition cell. Since each Q-value prediction comes from a single linear model, the prediction can be explained by the feature weights of the model.

This approach, has been evaluated towards mimicking a DQN model, and compared against CART, M5 with regression tree, Fast Incremental Model Tree (FIMT) baseline and with Adaptive Filters (FIMT-AF). The evaluation environments include Mountain Car, Cart Pole and Flappy Birds all with discrete actions but with discrete or continuous states. Evaluation concerns approximating the Q values in DQN, reporting on the number of the tree leaves constructed. Furthermore, the sampling efficiency of LMUT is evaluated by computing the correlation and testing error of LMUT, as more transitions are provided. Interestingly, LMUT models are evaluated in terms of the Average Reward Per Episodes by performing the tasks (e.g., playing games) in a direct manner: It is reported that LMUT among other mimic models achieves performance which is closest to the DQN. As far as interpretability is concerned, authors propose (a) a way to measure splitting features’ ”importance” using the total variance reduction of the $Q$ values, (b) a way to extract and present rules in the form of partition cells, each cell representing a range in specific splitting features, and (c) a way of highlighting pixels (when DRL takes raw pixels as input) while explaining local decisions. However, these options are not evaluated in any setting with human subjects.

Concluding, this is a mimicking approach, where the interpretable model is trained with transitions caused by the interactions of the DRL agent with the environment, also consulting DRL model decisions and values, via the function $f(\cdot)$. The interpretable model represents the agent policy, and as mentioned above, it can be exploited in several ways to provide interpretations. The article does not suggest any particular function $\epsilon$ implementing an explanation logic.

To address the need of explaining sequences of decisions, N. Topin and M. Veloso in (PolicyLevelExplanations, ) introduce Abstracted Policy Graphs (APGs). APGs are Markov chains of abstract states and are assumed interpretable representations of policies, concisely summarizing them, so that individual decisions can be explained in the context of expected future transitions. The authors propose the APG Gen method for generating APGs for deterministic policies given a learned value function and a set of observed (potentially off-policy) transitions.

An APG is a Markov chain over abstract states where edges are transitions induced by actions from the original MDP. The basic assumption here is that, if the agent’s transitions between grounded states are approximated using a Markov chain between abstract states, then grounded states which are treated similarly (thus, can be treated interchangeably) are readily identified (i.e., in groups) and the agent’s transitions between abstract states can be predicted. Interchangeable ground states grouped into abstract states should have similar future outcomes in terms of rewards, which are assumed known, and the importance of all features for a set of similar states should be low. The importance of a feature is calculated by the Feature Importance Ranking Measure (FIRM), as the variance of the conditional expected score of a state $s$ for that feature, with respect to an arbitrary function $g(s)$. This score is the average value of $g(s)$ for all states, where that feature takes a specific value $v$.

APG Gen (PolicyLevelExplanations, ) uses states’ similarity based on actions taken under the policy, to repeatedly divide abstract states along important features, and then computes transition probabilities between them. Important features can be trivially recorded for each abstract state, allowing for humans to inspect which features affect agent’s decisions, also providing a summary of an abstract state and of the corresponding ground states.

APG is evaluated in an abstraction of a production task for a multi-component item using a number of manufacturing steps, in deterministic and stochastic settings. Policies are computed using value iteration. In providing local explanations, APG is evaluated in terms of the features’ prediction accuracy (important vs. not important) as a function of the portion of states sampled: This provides a view on how APG generalizes on features’ importance, beyond the states seen. Additionally, n-hop action predictions are evaluated to measure the error caused by assuming that only single actions are performed for a transition between abstract states. This is measured by the action prediction agreement (predicted vs actual) for increasing time horizon. Finally, the size of “explanations” are measured by the number of abstract states (nodes in APG) against the number of grounded states in the MDP (reported sub-linear). No evaluation with human subjects in simulated or real-world settings are reported.

Concluding, APG Gen is a mimicking approach, with the objective of providing an interpretable model of a learned policy, considering a trained policy model. That model is realized by an APG. APG Gen takes as input transition samples $(s,a,s^{\prime})$ from the learned policy, the value function, as well as features’ importance. APG is the interpretable model, and local explanations may be provided consisting of the features that are important for a specific state. Reference (PolicyLevelExplanations, ) does not propose an explanation function $\epsilon$ for providing interpretations’ surface representations.

PIRL (Programmatically Interpretable Reinforcement Learning) is an RL framework (PIRL, ) designed to generate agent policies that are represented using a high-level, domain-specific programming language. Such programmatic policies are not as performant as those from DRL methods, but are interpretable and being amenable to verification by symbolic methods. A. Verma et al. in (PIRL, ) propose the Neurally Directed Program Search (NDPS) method for solving the challenging non-smooth optimization problem of finding a programmatic policy with maximal reward, minimizing the distance from an “oracle” DRL policy.

Specifically, considering a Partially Observable Markov Decision Process (POMDP) setting, a functional language, and given a policy sketch that syntactically defines a set of programmatic policies, the objective is to find a program in this set with maximal long-term reward. To restrict the search, the NDPS algorithm performs directed policy search guided by an oracle policy learned by a DRL method, via local search. NDPS measures the “distance” between the outputs of a programmatic policy and the outputs of the oracle policy, given a set of “interesting” histories which are enriched with additional histories generated by the programmatic policy (a technique that in (PIRL, ) is called “input augmentation”). Further optimization improves the programmatic policies found (e.g., being shorter).

PIRL is evaluated on the task of learning to drive a simulated car in the TORCS car-racing environment, using as oracle a policy computed by the Deep Deterministic Policy Gradient (DDPG) algorithm: NDPS policies are compared to human-readable policies, also showing that PIRL policies can have smoother trajectories, and better generalization abilities compared to the policies discovered by DRL. No evaluation of the explainability or interpretability of PIRL is provided, with or without human subjects.

Concluding, PIRL with NDPS is a mimicking process, exploiting agents’ interactions with the environment to find optimal interpretable programmatic policies that are presented as functional programs.

INTERPOLE is an imitation learning method that jointly models the agent’s policy (belief-action mapping) and belief-update process, towards interpreting demonstrated behaviour. As such, it can be used to model any “expert” demonstrated policy, without assuming unbiasedness of agent’s beliefs or optimality of policies. In so doing, A. Hüyük et al. in (ExplainingByImitation, ) propose an inherently interpretable imitation learning method, locating factors that contribute to individual decisions, in a language that is readily understood by domain experts, accommodating partial observability and operating offline.

INTERPOLE aggregates sequential observations through agent’s subjective belief-update process (decision dynamics) and probabilistic belief-action mapping for determining actions (decision boundaries). The objective is to model the most likely parameterizations $\theta=(T,O,b_{1},\eta,\{\mu_{a}\}_{a\in A})$, drawn from some prior $\mathbb{P}(\theta)$, i.e. the likelihood of $\theta$ with respect to a given set of demonstrated observation trajectories $\mathcal{\bar{D}}$ and unobserved state trajectories $\mathcal{D}$: $T(s_{t+1}|s_{t},a_{t})$ is the state transition probability, $O(z_{t}|s_{t},a_{t},s_{t+1})$ is observation’s probability after a transition to a new state, $b_{1}(s)$ is the probability of being in state $s$ in $t=1$, the smoothness of transitions between decision boundaries is tuned by the temperature factor $\eta$, and $\{\mu_{a}\}_{a\in A}$ are actions’ mean vectors inducing decision boundaries over the belief space. Parameters $\theta$ are jointly determined by means of an expectation-minimization (EM)-like algorithm for maximizing $\mathbb{P}(\mathcal{D})$, given that $\mathcal{D}$ is not known. Estimating jointly the parameters $\theta$, INTERPOLE learns agent’s decision and belief dynamics (not the environment dynamics) offering the most plausible explanation of how the agent reasons.

INTERPOLE has been evaluated using simulated and real world demonstrated trajectories for disease diagnosis, focusing on interpretability, accuracy of learned policies, and subjectivity towards recovering interpretations of behaviour even if the agent is biased. For interpretability, INTERPOLE has been evaluated by nine clinicians focusing on the utility of representing possibly subjective action-belief trajectories instead of raw action-observation trajectories, and on understanding policies by means of suboptimal decision boundaries, instead of using a reward function: The majority of clinicians preferred the explanations offered by INTERPOLE. Regarding accuracy, INTERPOLE is evaluated in modelling the belief-update process and the belief-action mapping (policy), learning high quality models driving agent’s subjective reasoning.

INTERPOLE (ExplainingByImitation, ) addresses the policy explanation problem, offering a method for imitating agents’ possibly suboptimal policies without assuming unbiasedness beliefs and objective reasoning. The interpretable model $c_{g}$ comprises decision dynamics and decision boundaries and is specified by $\theta=(T,O,b_{1},\eta,\{\mu_{a}\}_{a\in A})$ without assuming a fully observable setting, given a set of demonstrated action-observation trajectories $\mathcal{\bar{D}}$. The explanation logic $\epsilon$ visualizes a belief simplex in low dimensional spaces, showing decision boundaries and belief trajectories.

With the aim to assist human experts in decision making, J. Skirzyński et al. in (InterpretablePlanningStrategies, ) introduce the Adaptive Imitation-Interpretation (AI-Interpret) algorithm for transforming policies into sets of interpretable descriptions by means of decision rules that have a disjunctive normal form (logical program policies), presented as flawcharts.

Specifically, given a set of demonstrated trajectories $\mathcal{D}$ produced by the original policy, a domain specific language specifying the predicates for constructing the logical program policies, a parameter $\alpha$ defining the threshold at which the discovered policy (represented by a logical formula) is at least better than the original policy, and the maximum rule length ($d$) of the logical formula, the objective is to find the logical formula that maximizes the number of demonstrations in $\mathcal{D}$ that it can imitate w.r.t. $\alpha$ and $d$. AI-Interpret uses the predicates to separate the set of demonstrations into clusters. Clusters (whose number is determined automatically) are evaluated by means of a heuristic value that shows the similarity of trajectories in the cluster and how representative the cluster is. Doing so, the algorithm can consider increasingly smaller sets of demonstrations (disregarding less important clusters) and employ Logical Program Policies (LPP) (LPP, ) in a structured search searching for the simplest logical formula w.r.t. $\alpha$ and $d$ that imitates most of the demonstrated trajectories. Finally, the formulae are transformed into decision trees, and then visualized as flowcharts that people can follow to execute the strategy.

AI-Interpret is evaluated in three types of challenging planning tasks. The central question is whether discovered flawcharts can support decision-makers in the process of planning. To achieve that, authors in (InterpretablePlanningStrategies, ) perform one large behavioral experiment for each of the three types of tasks and a fourth experiment in which they evaluate whether AI-Interpret improves human decision-making against conventional training (i.e., by providing feedback). Results show that the proposed approach (a) allows people to largely understand the automatically discovered strategies and use them to make better decisions, and (b) is more effective than conventional training, improving human decision-making.

Overall, AI-Interpret (InterpretablePlanningStrategies, ) addresses the policy explanation problem, discovering policies described by means of logical programs. This is a mimicking approach, as the input it requires are demonstrations from the closed-box RL policy model. The interpretable model $c_{g}$ is the LPP discovered, and the explanation logic $e_{g}$ transforms LPP to flowcharts.

S. H. Huang et al. in (CommunicateObjectives, ) conjecture that, end-users, to understand and predict what the agent will do, need to have an accurate mental model of the agent’s objective function. To reduce the time needed for this otherwise time-lengthy process, authors propose to model how people infer objectives from observed behavior, in order agents to demonstrate behaviors that are maximally informative.

Following an algorithmic teaching approach, authors in (CommunicateObjectives, ) define candidate models of human inference regarding robot’s objective function, given robot’s optimal behavior in specific environments: These demonstrations increase the probability of humans to infer the correct objective function. The objective function is assumed to be a linear combination of (possibly complex functions on) environment features in $M$, weighted by parameters $\phi^{*}$. Features are supposed to be known by humans (although not always the case), who need to learn the true reward parameters. This involves searching for a sequence of environments $E_{1:n}$, such that when the person observes the optimal trajectories in these environments, their updated belief places maximum probability on the correct reward parameters. The challenges tackled to solve this optimization problem are as follows: (a) Humans are likely to be approximate in their inference, while (b) the agent needs to incorporate a model of this approximate inference, (c) encouraging full coverage of all possible strategies that the agent is capable of adopting. Updates of humans’ beliefs for the reward parameters can be modelled either via an inverse reinforcement learning technique (exact method), or via Bayesian inference (approximate method), modelling how probable humans consider trajectories to be, given their beliefs on the reward function. To address the coverage challenge, a new term is added to the formulation of the optimization problem, ensuring that no extra trajectory examples are selected unless these examples increase significantly the probability of inferring the true reward parameters. Extra examples are chosen to be the best with respect to the best approximate model.

Experimental results evaluate how approximate and exact inference models perform in a driving simulation environment. The aim is the system to provide to participants examples of how the car drives, with the goal of being able to anticipate how it will drive when they ride in it. The analysis of models is done with ideal users and with human subjects. Analysis shows that while not just any approximate-inference model is better than the exact one, choosing a suitable approximate model helps significantly. The user study on coverage concludes that coverage with the right model of approximate inference have a significant advantage over RL models assuming exact inference users.

Concluding, reference (CommunicateObjectives, ) provides methods that address the objective model explanation problem, focusing on teaching the agent’s reward function true parameters via examples. The proposed approach explicitly incorporates a model of the human beliefs on reward parameters and aims to provide a view on the true objective function. While the interpretable model can be considered to be the linear objective function, no explanation logic is provided here.

H. Yau, C. Russell and S. Hadfield in (WhatDidYouThink, ), aim to interpret decisions by describing agents’ intended outcome. This method applies to RL methods that estimate values off/on policy, given discrete states and actions and a deterministic MDP. The approach is demonstrated to off-policy $Q$-learning and aims to recover the future events that impact the $Q$-values learned.

This approach learns a belief map of discounted expected sum of state visitation H. H is updated when a Q-value is updated, as follows: $\textbf{H}(s_{t},a_{t})=\textbf{H}(s_{t},a_{t})+\alpha(1_{s_{t},a_{t}}+\gamma\textbf{H}(s_{t+1},argmax_{a}Q(s_{t+1},a_{t+1}))-\textbf{H}(s_{t},a_{t}))$, where $1_{s_{t},a_{t}}$ denotes an indicator function with 1 if the agent is at $(s_{t},a_{t})$ and 0 otherwise.

Given a deterministic reward function $R_{s_{t},a_{t}}$, the belief map H is consistent with $Q$ if $vec(\textbf{H}(s,a))^{T}vec(R(s,a))=Q(s,a),\forall(s,a)\in S\times A$. Authors prove that if H and $Q$ are zero-initialized, then H and $Q$ are consistent for all iterations of the algorithm in the tabular case. For deep Q-learning methods, there is no guarantee for consistency of H with Q.

The proposed approach is evaluated in domains with deterministic or stochastic reward functions and with continuous or discrete state-action space. In environments with stochastic rewards and continuous states, new discrete states are added, or states are discretized. The method is demonstrated to work effectively for tabular and deep Q-learning agents.

This is a generic approach that addresses the objective model explanation problem, providing projections of future intended trajectories of agents. The interpretable model is provided by the belief maps, and explanations are provided by means of belief maps visualizations showing the value of each intended future state-action pair. The method is limited to low-dimensional state-action spaces.

P. Madumal et al. in (DistalExplanations, ) emphasize on the role of opportunity chains for explaining agents’ behaviour, and introduce a distal explanation model that can generate opportunity chains as explanations for model-free RL agents. An opportunity chain takes the form of “A enables B and B causes C”, where B is the distal event or action, and can inform the explainee about long term dependencies between events, when certain events enable others.

Reference (DistalExplanations, ) proposes a distal explanation model that learns opportunity chains using a recurrent neural network (RNN). As distal explanations by themselves would not make a complete explanation, this approach uses action influence models (CausalLens, ) that approximate the causal model of the environment relative to actions taken by an agent, to get the agent’s goals. It further improves upon action influence models (specifying the effects of actions in features) using decision trees instead of structural equations used in (CausalLens, ) (presented in Section 4.4.7) to represent the agent’s policy. The decision tree policy model is trained concurrently with the RL policy model, assuming a model-free algorithm and exploiting state-action samples using an experience replay buffer. The tree is constrained to have a max number of leaves, equal to the agent actions. The same dataset of samples exploited for learning the decision tree model is exploited by a prediction RNN to approximate distal actions and their cumulative rewards.

The accuracy of distal prediction and counterfactuals is evaluated in six benchmark domains using different model-free RL algorithms. Benchmarks have a mix of complexity levels and causal graph sizes (number of actions and state variables). Results show that the distal explanation model is robust and accurate across different environments and algorithms. Furthermore, experiments with humans using RL agents report on task prediction (i.e., understanding of the agent) and subjective explanation satisfaction. Agents are trained to solve 1) an adversarial task; 2) a search and rescue task; and 3) a human-AI collaborative build task; all in Starcraft II. The results obtained for explanation quality show that distal explanations perform substantially better than baselines in human-agent collaborative tasks.

Concluding, this is a mimicking process, where the interpretable models (decision tree and opportunity chains) are trained by exploiting RL agent’s interactions with the environment. The interpretable models comprise a policy model $c_{g}$ using a decision tree, and a causal model representing the objectives ($c_{o}$), as well as the importance of specific responses using opportunity chains. The explanation logic $\epsilon$ exploits models to produce explanations in natural language.

The approach proposed by Z. Juozapaitis et al. in (RewardDecomposition, ) decomposes rewards into sums of semantically meaningful reward types, so that actions can be compared in terms of trade-offs among the types, with a focus on explainability. In particular, this work introduces the concept of minimum sufficient explanations for compactly explaining why one action is preferred over another in terms of the reward types.

Specifically, assuming that the MDP formulation incorporates a set of reward components/types $C$ and defining a vector-valued reward function with a component for each $c\in C$, $R_{c}(s,a)$, the agent objective is to optimize the overall (mixed) reward function $R(s,a)=\sum_{c\in C}R_{c}(s,a)$. This vector-valued reward allows for defining a vector-valued $Q$-function, with action-value components $Q_{c}(s,a)$ for rewards of type $c$. The overall Q-function is defined to be $Q(s,a)=\sum_{c\in C}Q_{c}(s,a)$. This results in a straight-forward heuristic tabular algorithm updating $Q$ values in a standard way for each of the components, called HRA. The definitions are extended to DRL settings, where each $Q_{c}$ is parameterized by means of $\theta_{c}$. This results into DRL algorithms for decomposed reward (dr), drDQN and drD-SARSA.

To gain insight into why an agent prefers action $a_{1}$ over $a_{2}$ in state $s$, i.e., $Q(s,a_{1})>Q(s,a_{2})$, this approach uses the difference explanation (RDX) as the difference of the decomposed $Q$-vectors. RDX is exploited to find the minimal sufficient explanations $MSX^{+}$ and $MSX^{-}$ for each pair of actions and state, indicating the “critical” positive and negative reasons (in terms of reward types), respectively for the preference.

Case studies in two environments illustrate the potential of visual explanations in the hands of an RL practitioner. Authors show how decomposed reward methods support spotting certain “bugs” in the agent’s action values and even help identify a more subtle issue related to the interaction of the gradient optimizer and the DRL loop.

Concluding, this approach addresses the outcome explanation problem, focusing explaning decisions on actions against other actions locally, i.e., in specific states, exploiting reward types. The interpretable model $c_{l}$ takes the form of tuples $(MSX^{+},MSX^{-})$ per state and applicable action couples, exploiting (decomposed) action values provided by $f(\cdot)$. The explanation logic $\epsilon_{l}$ is implemented by means of bar charts on features.

The reward decomposition approach has been proposed as the basis for a set of decision-making tasks in (StrategicTasks, ), in environments that naturally provide multiple reward signals: The goal is to enhance explainability by providing high-level abstractions for sequential tasks.

B. Hayes and J. A. Shah in (AutonomousPolicyExplanation, ) aim to enable an autonomous agent to reason over and answer questions about its underlying control logic $L$, independent of its internal representation. First, it learns a domain model of the agent’s operating environment from real or simulated demonstrations, exploiting programmer specified code annotations for actions, and variables (state parameters/attributes) affected by actions. Within this learned domain model, this approach uses statistics computed over data extracted from continued observations to construct a behavioral model that approximates the agent’s control logic. The article shows applicability of the proposed approach in tabular Q-learning, deep Q-learning and hand-coded controller logic.

To explain the policy $L$ given a query $q$, this work provides an interpretation function $c$ to provide a natural language response $Response$. $L$ is constrained to state parameters and annotated functions. $Response$ is constrained to a template-based approach using natural language.

The interpretation function $c$ is defined to be $c=Summarize\_attributes\textit{ o }Resolve\_states\textit{ o }Identify\_question$, where, $Resolve\_state$ resolves the question identified by $Identify\_question$ to a set of problem states (context), and $Summarize\_attributes$ summarizes attributes of problem states into a comprehensive representation. The $Resolve\_state$ realizes the function $f(\cdot)$, providing a representation of relevant parameters to resolve a question. Depending on the question type (see below), this may take the form of searching and gathering state parameters’ from states in a region, or from similar states given a specific state, or from near-by states. Questions concern when an agent will behave in a specific way, how it will behave under specific conditions, and deviations from the expected behavior. Question-answering is template driven using a response resolution algorithm per query type. Depending on the question type, $c$ provides an explanation for a specific response (i.e., locally) at the granularity of policy actions, explaining the control logic in specific contexts (i.e., sets of states). The explanation logic $\epsilon$ is implemented by the $Compose\_summary$ function and is realized by means of Boolean classifiers using communicable predicates, which are similar to STRIPS-style planning predicates, associated with natural language descriptions.

This work allows human collaborators to shape their expectations on agents’ behaviour without requiring a full understanding of an agent’s logic, and facilitates the debugging of aberrant behaviors by explaining differences between the conditions under which particular actions occur. The article demonstrates the applicability of the proposed method within three representative robotics domains with discrete and continuous state parameters, in single/multi agent settings, using different types of controllers. However, experiments focus on the applicability of the method, rather than on the accuracy/fidelity and conciseness of the explanations provided in any specific human-robot collaborative setting.

N. Wang et al. in (TransparencyCommunicationforRL, ), as a continuation of previous work on static models (TransparencyCommunicationforRL, ), discuss the design of model-based and model-free RL for robots in a human-robot simulation testbed. The goal is to provide communications for the robot’s decision making update process towards interpretability and repairing trust. This is achieved by interpreting components of the robot’s decision making and learning process, assuming that the robot bases its decisions on a POMDP model.

Considering a dynamic POMDP and assuming a model-based RL that updates the POMDP model, the proposed method interprets a static POMDP after each update, informing the teammate about changes. This work focuses on updating the observations model, i.e., the probability for the robot to get any observation. Additionally, assuming a model-free RL, $Q$ values account for changes in a possible model, but with no explicit representation of these changes and model. To address this last issue, authors propose finding a potential POMDP that is consistent with the policy arrived at by the model-free RL. To do that, they compute the optimal policy for that POMDP exploiting the assumption of a piecewise-linear model (PieceWiseLinearFunctions, ), representing that policy by means of a decision tree, which is compared to the RL policy. If it matches, this POMDP is used to provide explanations.

This article does not report on the efficacy of explanations with human-subjects. As authors also point out, this must be done with different permutations of explanations’ aspects, also providing insight into what aspects should be included in the explanations and how to present them.

Concluding, this approach provides interpretations of POMDP components, with no access to the learning process itself. We can consider that functions $f(\cdot)$ provide updates on agents’ beliefs, transition models, or updates in the $Q$ values, while the interpretation process updates the existing POMDP, or searches for a potential matching POMDP, which serves as the interpretable model. Function $\epsilon$ provides explanations on POMDP components’ updates using natural language static templates. In addition to revealing information about individual POMDP components, templates can be created informing about the overall model and how the learning arrived at that model, also leveraging the interpretable policy models generated by exploiting the piecewise-linear assumption.

R. Iyer et al. in (TransparencyandExplanationInDRL, ) report on the interpretability of DRL Networks (DRLN) in visual tasks, proposing the Object-sensitive DRL (O-DRL) method that (a) incorporates explicit object recognition processing into DRL models, (b) forms the basis for the development of object saliency maps, and (c) can be incorporated in any existing DRL framework such as DQN and A3C. The goal here is to produce intelligible visualizations of agents’ state and behaviour, focusing on objects saliency, i.e., on the influence of objects features in agents’ decisions.

The proposed O-DRL approach uses a deep neural network (CNN) that takes as input object channels and original images to predict $Q$-values. Object channels represent types of objects recognized in an image and incorporate objects’ features. To compute object saliency, this proposal masks the object and computes the $Q$ values on the new (perturbed) state, and the difference of new values to the $Q$ values in the original state.

Authors report on experiments with humans, regarding the behaviour of a Pacman agent. The goals of the experiments were to (a) test whether object saliency maps contain enough information to allow humans to match them with corresponding game scenarios, (b) test whether participants could use object saliency maps to generate reasonable explanations of the behavior of the Pacman and (c) test whether object saliency maps allow participants to correctly predict the agent’s next action. Experiments include (a) a matching task, where participants match saliency maps to agents’ behaviour; and (b) a movement prediction task from video clips with or without saliency maps. Results show that object saliency maps can be linked to corresponding game scenarios by participants (thus, they do provide a basis for visual explanations) but, as results from the prediction task suggest, the exact use of salient maps must be further investigated in conjunction to providing rich contextual information about each situation.

This is an approach that addresses the outcome explanation problem in visual tasks, focusing on the influence of objects in agents’ decisions in specific states. The proposed interpretation method is realized by the O-DRL method that exploits $Q$ values, provided by function $f(\cdot)$ to rank importance of features (image pixels or objects). Interpretable models are realized by pixels’ and objects’ salience. The explanation function $\epsilon_{l}$ realizes surface representations of interpretations using visual means.

Recognizing that perturbation-based approaches for RL, as that proposed by S. Greydanus et al. in (VisualizingAtariAgents, ) and the O-DRL approach (TransparencyandExplanationInDRL, ) in 4.4.4, tend to produce saliency maps that are not specific to the action of interest, N. Puri et al. in (ExplainYourMove, ) propose SARFA to generate saliency maps that focus on explaining the specific action decided, balancing between specificity and relevance.

SAFRA is a pertrubation-based approach for generating saliency maps focusing on capturing the impact of pertrubation only on the $Q$ value of the action to be explained (specificity), and downweighting features that alter the expected rewards of actions other than the action explained (relevance): Specificity is measured by capturing the relative change to the $Q$ value for the action to be explained w.r.t. other actions, and relevance by considering how the relative expected reward of taking some actions (other than the one explained) changes with a pertrubation. The harmonic mean of these two measures produces the salience of state features.

SAFRA in (ExplainYourMove, ) has been evaluated qualitatively in games (chess, Attari games and Go), showing that in comparison to O-DRL (TransparencyandExplanationInDRL, ) and the approach in (VisualizingAtariAgents, ) produces more focused saliency maps. In addition, human studies on problem-solving chess puzzles show that saliency maps produced by SAFRA are less confusing than those produced by (TransparencyandExplanationInDRL, ) and (VisualizingAtariAgents, ). SAFRA is compared to the other approaches using a chess saliency dataset produced by human experts, showing superiority in identifying salient features and robustness in pertrubations.

Overall, SAFRA (ExplainYourMove, ) addresses the outcome explanation problem, focusing on visual tasks and aiming to identify state features whose salience is specific and relevant to the decided action. SAFRA exploits $Q$ values, provided by $f(\cdot)$ and interpretation is provided by the salience of state features. The explanation is surfaced using visual means, showing the salience of features.

Regarding the power of salient maps to assess the degree to which hypotheses about features of the learned policy correspond to the semantics of RL settings, A. Atrey et al. in (ExploratoryNotExplanatory, ) suggest that salience maps should be used as an exploratory rather than as an explanatory tool and propose a methodology grounded in counterfactual reasoning, focusing on issues of subjectivity, unfalsiability and cognitive biases on mapping salient features to semantic concepts and behaviours. In addition to that, saliency maps have been used also in combination with policy explanation methods in (LocalAndGlobal, ), investigating their role when put in the context of agents’ strategies (discussed in Section 4.2.1).

Y. Fukuchi et al. in (AutonomousSelfExplanation, ) propose Instruction-based Behavior Explanation (IBE) to explain agent’s future behavior within a sufficient temporal extent, while the policy is being learned in an interactive RL setting. In IBE, an agent exploits the instructions given by a human expert (which may not indicate the actions to be performed) to explain its own behavior: This happens under the assumption that when the agent receives more rewards it is likely that it followed the instructions given.

IBE consists of two steps: (i) estimating the target of the agent’s actions by simulation: the target is the change in the environment state after a fixed temporal extent for explanations specified by agents’ actions in $n$ steps, and (ii) acquiring a mapping from the target to the instructions, through a k-means classifier, in order to explain the action target based on the instruction signal given by a human expert. Authors report on experiments with humans in a game setting, using two DQN agents, each at a different training stage. The most important finding is that the environment changes must be chosen deliberately for assigning an explanation signal: The parameter $n$ that defines the number of agent’s actions to a target state can not be fixed, while predictions in settings of high-complexity must be accurate.

IBE (AutonomousSelfExplanation, ) provides a local interpretation method, interpreting the decision of the agent in specific states, given a target state in a constant temporal extent of $n$ time steps. The interpretation function is the IBE method itself, exploiting agents’ target and instructions provided by experts. Thus, the RL method is treated as a dark box, where the function $f(\cdot)$ fetches instructions provided by experts, provided as representations of agent’s behaviour. The classification of assessed targets to instructions provides a local interpretable model. The explanation logic is context and domain dependent: examples of visualisations are provided in (AutonomousSelfExplanation, ). As stated in (AutonomousSelfExplanation, ), prediction of changes is challenging in complex settings, and the temporal extent may need to vary in different contexts, even for the same agent.

P. Madumal et al. in (CausalLens, ) aim to generate explanations using causal chains from action influence graphs, for model-free RL agents. This is accomplished by encoding causal relationships between variables of interest and learning a structural causal model (StructuralCausalModel, ) during RL. Action influence models approximate the causal model of the environment relative to actions taken by an agent. This approach uses causal models to generate contrastive explanations for why and why not questions, and formulates the minimally complete explanations, and minimally complete contrastive types of explanations to be provided from an action influence model.

A critical part of this approach is learning the structural equations, i.e., the quantitative influences of actions on variables: Assuming that a DAG specifying causal direction between variables is given, structural equations are approximated as multivariate regression models during the training phase of the RL agent, exploiting samples of agent interaction with the environment via an experience reply. Therefore, this is a mimicking process, where interpretable models are structural equations specifying the effects of actions in features, the causes of rewards, and the states resulting in rewards. These local models $c_{l}$ provide local response and local objectives’ explanations. The explanation logic $\epsilon$, provides explanations to the two types of questions: “why” and “why not”.

This approach has been computationally evaluated on six RL benchmarks domains using six different model-free RL algorithms with discrete actions. Results indicate that the proposed models are robust and accurate enough to perform task prediction with a negligible performance impact. A human study (120 participants) evaluated the method in task prediction, explanation satisfaction, and trust, showing that the proposed model performs better than the tested baseline, but its impact on trust is not statistically significant.

S. Wenjie et al. in reference (Self-SupervisedDiscoveryOfCausalFeatures, ) propose a self-supervised interpretable framework (SSINet) to explain a trained policy model of vision-based RL agents to locate fine-grained causal features towards gathering state-specific and task-relevant evidence for the agent’s decisions.

Specifically, given a trained policy model, the proposed framework learns an explanation model that predicts an attention mask. The basic attention patterns learned are exploited for identifying the relative importance of features and analysing failure cases. If the generated actions are consistent when the policy takes as input either the state or the attention-overlaid state, the features highlighted by the proposed method are considered to be task-relevant and constitute evidence for the agent’s decisions. The agent in (Self-SupervisedDiscoveryOfCausalFeatures, ) is pre-trained using model-free RL algorithms including proximal policy optimization (PPO), soft actor-critic (SAC) and twin delayed deep deterministic policy gradient (TD3).

More technically, the SSINet uses a U-Net architecture, providing “masked” states through a feature extractor, a mask decoder and a sigmoid non-linearity. SSINet uses the same feature extractor with the pretrained agent policy model, thus SSINet trains only the mask decoder. SSINet is trained using a dataset of state-action pairs generated by the policy. Training is “driven” by a mask loss function aiming to satisfy two desiderata: Maximum behaviour resemblance, i.e., the agent’s behaviour using masked states must be as consistent as possible with that when using original states, and minimum region retaining, i.e., attentions must attend as little information as possible.

SSINet has been evaluated on several Atari 2600 games, as well as on Duckietown, a self-driving car simulator environment. Empirical results verify the effectiveness of the proposed method, and demonstrate that the SSINet produces high-resolution and sharp attention masks to highlight task-relevant information. Special emphasis has been given to explaining why the agent performs well or badly quantitatively, by introducing evaluation metrics that assess the effectiveness of attention masks, in multiple RL algorithms and actor architectures.

In conclusion, the SSINet proposed in (Self-SupervisedDiscoveryOfCausalFeatures, ) is a self-supervised machine learning process for vision-based DRL agents. Although it exploits samples from a trained policy model, it does not aim to produce an interpretable model that mimicks or that distills the knowledge acquired in the policy model: It rather addresses the outcome explanation problem, providing a local model $c_{l}$ that identifies salient state features, which are further presented via appropriate visual masking techniques realized by $e_{l}$, applied in the original state.

Y. Coppens et al. in (DistillingDRLInSDTs, ) illustrate how Soft Decision Trees (SDT) (DistillingNNsInSDTs, ) can be used as interpretable policy models. SDT are hybrid classification models of binary trees of predetermined depth, and NNs. Each branching node represents a hierarchical filter that influences the classification of input data. The leaf nodes learn softmax distributions over possible classes using model parameters $\theta^{SDT}$, while the overall loss is minimized using mini-batch gradient decent optimization.

Using state action pairs generated from the policy learned using the actor-critic Proximal Policy Optimization (PPO) algorithm, the soft decision tree is trained to classify states to actions. The applicability of the method is evaluated on the Mario AI benchmark: Authors show the fidelity with which SDTs can provide interpretations, and elaborate on the performance of these models when one would consider to use SDTs directly for task execution.

This is a mimicking process (as authors in reference (DistillingDRLInSDTs, ) also state), despite the article title that characterizes the approach as a distillation process. The model $c_{l}$ is the soft decision tree, which provides local only interpretations, classifying states to agent actions. Despite the possibility of examining the learned filters along any decision tree path from the root to leaf, it is questionable whether the soft decision tree offers a fully interpretable model, given the parameterized leaf nodes and the decision tree filtering nodes leading to an action distribution. As noted in the article, since the distributions in the leafs need to generalize and provide an action strategy for multiple state samples, it is to be expected that the resulting action distributions in the soft decision trees can differ from the NN output of the PPO agent for individual input samples: This results in low fidelity in mimicking the PPO policy. The interesting aspect of this work is that, bigger decision trees, offer greater agent rewards. However this comes with the cost of reduced interpretability. The explanation logic $\epsilon$ provides surface representations using hitmaps, exploiting the SDT model in spatial settings. The use of SDTs in non-spatial settings is something that worth further investigation.

F. Cruz, R. Dazeley and P. Vamplew in (MemoryBAsedXRL, ) propose a memory-based explainable reinforcement learning (MXRL) approach according to which the agent exploits an episodic memory to infer the probability of success and the number of transactions to reach the goal state. Specifically, the agent exploits past interactions with the environment, and by “introspection” can compare the probability of choosing an action against the probability of being successful, thus providing explanations in terms of the necessity to complete an intended task. The interpretable model aims to explain the agent’s decision for an action at a specific state, based on past experience, exploiting transitions in successful runs. It is assumed that all transitions have been recorded. This limits the method applicability to small, discrete state - action spaces, such as the grid world in which experiments have been conducted.

In essence, this is a distillation process where the agent distills knowledge regarding the reasons that the RL agent favors specific actions for specific states, through past experience gathered in an “interpretation” that applies at the episodic memory. The explanation logic is provided via hitmaps and diagrams, showing probabilities for choosing actions in states, and probabilities to succeed in reaching a goal state.

As noted in (Miller, ), people usually prefer explanations that focus on selected, specific evidence, instead of showing every possible cause of a decision. Based on this insight, authors in (LRP-argmax, ) aim to adjust the layer-wise relevance propagation (LRP) (LayerWiseRelPropagation, ) saliency map approach to focus on the parts of the input that are most relevant for the decision-making process of a DRL agent. Their contribution is twofold: (a) Their adjustment uses an argmax function to follow only the most contributing neurons, filtering out the most relevant information; (b) they adjust LRP to dueling DQN convolutional layers.

LRP describes a concept that can be applied to any classifier if that fulfills the following two requirements. First, it has to be decomposable into several layers of computation where each layer can be modeled as a vector of real-valued functions. Second, the first layer has to be the input $x$ of the classifier and the last layer has to be the real-valued prediction of the classifier $f(x)$. Any DRL agent fulfills those requirements, considering actions decided as the output: In this case LRP can be used to address the outcome explanation problem. According to the LRP concept, relevance values $R^{l}_{j}$ are assigned to each computational unit $j$ of each layer $l$ in such a way that $R^{l}_{j}$ measures the local contribution of the unit $j$ to the prediction $f(x)$. The calculation of $R^{l}_{j}$ values follow the LRP concept, if it sets the relevance value of the output unit to be the prediction $f(x)$ and calculates all other relevance values by defining $R^{l}_{j}=\Sigma_{k\in\{j\textit{ is input to }k\}}R^{l,l+1}_{j,k}$ for messages $R^{l,l+1}_{j,k}$, such that $R^{l+1}_{k}=\Sigma_{j\in\{j\textit{ is input to }k\}}R^{l,l+1}_{j,k}$. Relevance values $R^{l}_{j}$ can be computed in a backward pass, starting from the output layer, towards the input layer. This concept has been applied to the dueling DQN architecture in (LRP-argmax, ), specifying the backward pass through the fully connected and the convolution layers. However to find the most relevant input neurons contributing to a specific decision, authors propose the argmax rule, defining the messages as follows:

where $w_{i}$ are model weights and $a_{i}$ the inputs.

This approach has been evaluated on three Atari 2600 games to verify that the saliency maps generated are more selective than the ones created by alternative LRP methods, while still including the information expected from visual explanations: This can be beneficial to the transparency of DRL methods.

Overall, this is an approach that addresses the outcome explanation problem, aiming to provide the most salient features towards specific DRL agents’ decisions. The model constructed concerns the local contributions (relevance values) of inputs and neurons to the decision made, while explanations are provided visually by means of inputs’ saliency maps.