TDM: Trustworthy Decision-Making via Interpretability Enhancement

From self-driving cars to voice assistant on the phone, artificial intelligence (AI) is progressing rapidly. Though AI systems are undeniably powerful and continue to expand their role in our daily lives, emerging issues, such as lack of transparency and uncontrolled risk in decision-making, give rise to a vital concern: why should the end-users trust the decision support system? Reliance on technology and autonomy is strongly influenced by trust [1]. However, a recent study reveals that a mere belief in interacting with autonomous teammate led to diminished performance and passive behavior among human participants even though they were actually collaborating with a remote human partner [2], indicating that we do not trust the capabilities of AI systems as much as we trust humans.

Building trustworthy AI systems [3, 4, 5, 6] , therefore, becomes an important issue for humans to reap the full spectrum of societal and industrial benefits from AI, as well as for ensuring safety in the use of the system [7]. Indeed, in modern artificial intelligence systems, there are many interactive relations, primarily in the form of interactions between humans, between humans and the smart agents (human-computer interaction systems), between agents and the environment (interactive machine learning), and among smart agents (multi-agent systems).

It is also important to note that the trust is mutual. An AI agent must also gauge the trustworthiness of the end-users. However, very few studies exist in agent-to-human trust models. The recent Boeing 737 MAX 8 incidents [8, 9] have tragically shown the dangers of the lack of the automated agent’s ability to trust the end-users. The pilots of the crashed airplanes attempted to take control back from a malfunctioning flight control system, but the flight control system could not respond to the change of trust from pilots. Instead of trusting the pilots’ decisions, the flight control system continued to override the manual control resulting in fatal crashes.

Besides, AI systems, in general, need to comply with the norms of the social and cultural context if they are expected to operate side by side with humans. Therefore, trust is particularly important for normative agents that are expected to exhibit behavior consistent with the norms of a society or group. From the humans’ perspective, we trust things that behave as we expect them to, indicating that trust derives from the process of minimizing the perceived risk [10]. Perceived risk is often defined in terms of uncertainty about the possibility of failure or the likelihood of exhibiting improper behavior.

The typical approach to computationally modeling trust is through beliefs. In multi-agent settings, the agents maintain beliefs on the trustworthiness of other agents influenced by mutual interactions [11]. When end-users are involved, the agent maintains beliefs about the beliefs of humans’, concerning the trustworthiness of the agent [12]. It is difficult to design an explicit measure of trust without considering different perspectives. Recent work of Hind et al. [13] suggested several elements, such as fairness, robustness, and explainability, to increase trust in AI systems.

To open up the AI black-box and facilitate trust, it is critical to enable the AI system to be reliable, transparent, and explainable so that the system can achieve reproducible results, justify the decisions it makes, and understand what is important in the decision-making process. And this combination of features can be called “interpretability”. Deep learning algorithms have been successfully applied to sequential decision-making problems involving high-dimensional sensory inputs such as Atari games [14]. However, deep learning algorithms have limited interpretability and transparency, which gives rise to doubt by the human end-users due to a lack of trust and confidence in the AI systems. Prior research efforts [15] on interpretability often focus on explaining the results of learning when the underlying problem and architecture are complex. Symbolic planning [16, 17, 18, 19, 20] has been used to break down the complex task into simpler interpretable subtasks. However, existing studies [21, 22] focus on the performance gain of symbolic planning when it is applied to general sequential decision-making rather than interpretability.

Motivated by understanding the task-level behavior of the algorithm and elevating the trust between human end-users and the smart agents, we propose a unified framework of Trustworthy Decision-Making. Symbolic planning is utilized to perform reasoning and planning on explicitly represented knowledge, which results in task-level interpretability. We also introduce a computational trust metric based on the success ratio of symbolic tasks. The key insight is that the success is better measured when the tasks are interpretable. Achieving a reliable success rate of the tasks is equivalent to minimizing perceived risks of the agent’s behavior, thereby enhancing its trustworthiness.

The rest of the paper is structured as follows. Section II presents related research in trust, interpretability, followed by a review of prior trust evaluation studies in the extant literature from different perspectives. Section III introduces the background on symbolic planning along with an overview of the interactive sequential decision problem. Section IV provides a detailed description of the TDM framework. We introduce a learning algorithm facilitated by our formal framework in Section V. The optimality analysis for the converged plan generated by the algorithm is presented to substantiate the utility of our strategy. The experimental results of Section VI validate the interpretability of the high-level behavior of the system and the effectiveness of the trust metric while maintaining improved data efficiency. Section VII concludes by summarizing our results and delineating potential avenues of future research.

This section identifies and elaborates several key components of a computational framework for trustworthy decision-making, i.e., the definition of trust, interpretability and its relation to trust, and computational models of trust metrics.

To the best of our knowledge, there exists limited and inconclusive research on establishing a computational model of trust to measure the quality of a proposed task. In a quality-centric perspective, the quality of behavior should correspond to the plan’s trust score that indicates the level of trustworthiness. Such metrics need to be general enough to make evaluations across different tasks. A relevant research area in psychology is called intrinsic motivation, which is defined as the level of inherent satisfaction by accomplishing an activity. Interpersonal trust is strongly correlated to the trustor’s perceived intrinsic motivation of trustee [41, 42]. Combined with the role of interpretability in trustworthiness, this means a human is more likely to develop feelings of trust towards a smart agent if the human understands that the agent is intrinsically motivated to improve its abilities.

Different from extrinsic motivation, the inherent satisfaction is driven by an internal utility function [43]. This makes it a suitable general metric to make evaluations of different tasks. Intrinsically-motivated learning [44] used the framework of options. An option, or a subtask, can be initiated in some states to control the agent’s behavior in a subset of the environment with a terminating condition. Motivated by these factors, we propose a trust metric evaluation framework based on the internal utility function, which can be used to compute the trustworthiness values for the trust score and evaluate the high-level behavior of the system. One of the problems of interpretability is that it can induce unwarranted trust of unreliable agents [45, 46]. We hypothesize that the quality-based metric will alleviate this problem by justifying why the learned plans are good and trustworthy, whereas the abandoned plans are not, thus ensuring the agent’s learning process is reliable.

In this section, we introduce the motivation for developing a trust metric and the formulation of a symbolic planning strategy with respect to sequential decision making.

Consider a Markov Decision Process (MDP) ¹¹1We follow the style of notation in the 1st edition of reinforcement learning book by Sutton and Barto [48]. defined by a tuple $({\mathcal{S},\mathcal{A},P_{ss^{\prime}}^{a},r,\gamma})$. We will use the MDP to model the sequential decision-making problem in symbolic representation, where $\mathcal{S}$ is the set of symbolic states and $\mathcal{A}$ is the set of actions. Then $P_{ss^{\prime}}^{a}$ is the transition kernel that, given a state $s\in\mathcal{S}$ and an action $a\in\mathcal{A}$, defines the probability the next state will be $s^{\prime}\in\mathcal{S}$. Moreover, $r(s,a):\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$ is a reward function bounded by $r_{\max}$, and $0\leq\gamma<1$ is a discount factor. A solution to an MDP is a policy $\pi:\mathcal{S}\mapsto\mathcal{A}$ that maps a state to an action.

To evaluate a policy $\pi$, there are two types of performance measures: the expected discounted sum of reward for infinite-horizon problems and the expected un-discounted sum of rewards for finite horizon problems. In this paper we adopt the latter metric defined as $J^{\pi}_{\rm avg}(s)=\mathbb{E}[\sum\limits_{t=0}^{T}{{r_{t}}}|s_{0}=s]$. We define the gain reward ${\rho^{\pi}}(s)$ reaped by policy $\pi$ from $s$ as ${\small{\rho^{\pi}}(s)=\mathop{\lim}\limits_{T\to\infty}\frac{{J^{\pi}_{{\rm{avg}}}(s)}}{T}=\mathop{\lim}\limits_{T\to\infty}\frac{1}{T}\mathbb{E}[\sum\limits_{t=0}^{T}{{r_{t}}}]}.$ Gain reward ${\rho^{\pi}}(s)$ is often used to measure the lower bound of the expected cumulative rewards of a task w.r.t a given policy. For example, ${\rho^{\pi}}(s)$ is instrumental in deciding whether a specific task is rewarding enough to make the “stay-or-leave” decision [49]. More mathematical details of gain reward can be referred to [50, 51, 52, 53].

SP algorithms are also white-box algorithms. With the symbolic knowledge designed to be human-readable, the behavior of an SP agent that plans and reasons based on it is naturally interpretable. Prior works combine symbolic planning with other sequential-decision making algorithms to bring stability to plan-based learning [21, 22]. However, these works only highlight the performance improvement from the prior domain knowledge represented by SP.

Therefore, given action description $D$, here is an example of how to perform symbolic planning with action language $\mathcal{BC}$. Let $\langle s,a,s^{\prime}\rangle$ denote a transition from a symbolic state $s$ to a symbolic state $s^{\prime}$ by a set of action $a$. Then a planning problem can be formulated as a tuple $(I,G,D)$. The planning problem has a plan of length $l-1$ if and only if there exists a transition path of length $l$ such that $I=s_{1}$ and $G=s_{l}$. In the rest of the paper, $\Pi$ denotes both the plan and the transition path by following that plan. Given the above semantics represented in $\mathcal{BC}$, automated planning can be achieved by an answer set solver such as Clingo [58], and provide the solution to the planning problem.

We will also use the MDP to model the underlying sequential decision-making problem, define by a tuple $(\widetilde{\mathcal{S}},\widetilde{\mathcal{A}},\widetilde{P^{a}_{ss^{\prime}}},r,\tilde{\gamma})$, where $\widetilde{\mathcal{S}}$ consists of states of high-dimensional sensory inputs such as pixel images, $\widetilde{\mathcal{A}}$ is the set of primitive actions, and others are similarly defined as before. The goal is to learn both a sequence of subtasks and the corresponding sub-policies, so that executing the sub-policy for each subtask one by one can achieve maximal cumulative reward. It should be noted that our framework is not restricted to high-dimensional sensory inputs. However, we choose to focus on them since learning from such inputs is more challenging and relevant to real-world applications.

We assume human experts can provide a symbolic structure of the problem. The symbolic structure consists of the knowledge of objects, properties, and the preconditions and effects of executing subtasks in a given problem domain. Although this appears to be a lot of effort, it is possible to develop general-purpose action modules [59, 60, 61] as it has been shown that dynamic domains share many actions in common. Therefore, the symbolic formulation for one problem can be adapted to another with a little effort by instantiating a different set of objects or adding a few more rules. Our framework’s domain dynamics have a coarse granularity and high-level abstraction, enabling the decision-making process to be robust and flexible when facing uncertainty and domain changes.

With a symbolic representation given by the human expert, the TDM architecture is shown in Fig. 1. A symbolic planner generates a high-level plan, i.e., a sequence of subtasks, based on the trust evaluator’s evaluation feedback. We implement a mapping function to perform symbol grounding in our experiments, translating the problem domain to a symbolic representation. However, we assume such functions are generally available in different domains. After symbol grounding, the subtasks in the plan will be sent to the controller to execute so that the sub-policies for the corresponding subtasks are learned. Since it is possible that the controller cannot successfully learn the sub-policies, a metric is introduced to evaluate the competence of learned sub-policies, such as the success ratio over several episodes. After all subtasks in the plan have been executed, the trust evaluator computes the trust score by utilizing the trustworthiness values for subtasks. The evaluation for subtasks is returned to the symbolic planner and is used to generate new plans by either exploring new subtasks or sequencing learned subtasks that supposedly can achieve a higher trust score in the next iteration.

We present a learning algorithm for the TDM framework in Algorithm 1 and show TDM’s planning and learning process. Each episode starts with the symbolic planner generating a symbolic plan $\Pi_{t}$ for the problem $(I,G,D)$ (Line 4). The symbolic transitions of $\Pi_{t}$ are mapped to subtasks (Line 10) to be learned by a controller (Line 11). The controller performs deep Q-learning with $t_{e}$ using experience replay (Lines 12–15) to estimate the $Q$ value $Q(\tilde{s},\tilde{a};o)\approx Q(\tilde{s},\tilde{a};\theta,o)$, where $\theta$ is the parameter of the non-linear function approximator. The observed transition $(\tilde{s}_{t},\tilde{a}_{t},r_{e}(\tilde{s}_{t+1},g),\tilde{s}_{t+1})$ is stored as an experience in $\mathcal{D}_{o}$. The loss at $i^{\text{th}}$ iteration is calculated as an expectation over the collected experience and is given by

When the inner loop terminates, a symbolic transition $\langle s_{t},a_{t},s_{t+1}\rangle$ for the subtask $o_{t}$ is completed, and we are ready to update the trust score of the subtask (Line 18). Although various learning methods, such as Q-learning, may be used here, we use R-learning [62] to evaluate the trust score as an average reward case rather than a discounted reward case. With R-learning, the update rule becomes

The quality of $\Pi_{t}$ is measured by (2) (Line 20) and the trust score of the plan is updated according to $\hbox{\it quality\/}(\Pi_{t})$ (Line 21). The symbolic formulation is then updated with the learned  $\rho$ values (Line 22). With the symbolic formulation changed, an improved plan may be generated in the next episode. The loop terminates when no such improvement can be made.

The algorithm guarantees symbolic level optimality conditioned on R-learning convergence.

We use the Taxi domain [63] to demonstrate the behavior of learning and planning based on the trust score, Grid World [21] to show the ability to learn unmodeled domain knowledge, and on Montezuma’s Revenge [14] for interpretability and data efficiency. We use 1M to denote $1$ million and 1k to denote $1000$.

Since interpretability of the high-level behavior is critical to enhancing trust in a hierarchical sequential decision-making problem, we proposed the TDM framework in this paper by integrating symbolic planning with sequential decision-making operating on high-dimensional sensory input. We also presented a TDM-based deep learning algorithm. Deep learning architecture is used to learn low-level control policies for subtasks managed by high-level symbolic planning based on explicit symbolic knowledge. Both theoretical analysis and empirical studies on benchmark problems validate that our trust-score based algorithmic framework brings task-level interpretability to deep reinforcement learning and improved data efficiency induced by the symbolic-planning-learning framework of the agent.

There are several promising future work potentials in this research direction. For example, it would be intriguing to apply TDM in the human-computer interaction systems, allowing layperson to understand the system behavior and provide meaningful evaluation feedback to the machine. The second interesting direction is to apply the framework of TDM in multi-agent systems. Trust is important in interactions among different agents where the integrity of some agents is not always guaranteed. Interpretability may be used to help improve the behavior of the agents and lead to a better equilibrium.