Human Perceptions on Moral Responsibility of AI: A Case Study in AI-Assisted Bail Decision-Making

Theories of moral responsibility date back to Aristotle, who argued that an entity should satisfy both freedom and epistemic conditions to appropriately be ascribed to moral responsibility. Agents must act freely, without coercion, and understand their actions. Although recent scholarly work does not directly challenge these Aristotelian conditions, they argue that moral responsibility cannot be explained as a single concept, but that it involves a relatively pluralistic definition of what it means to hold someone morally responsible (Van de Poel, 2015; Shoemaker, 2011).

Scanlon (Scanlon, 2000) has proposed moral responsibility to be a bipartite concept. One is that there is an account of being responsible in rendering an agent worthy of moral appraisal. Another is that it is also possible to hold one responsible for specific actions and consequences. Expanding this bipartite concept, Shoemaker (Shoemaker, 2011) has proposed three different concepts of moral responsibility: attributability, answerability, and accountability. Various other definitions have been proposed (Van de Poel, 2015), including structured notions of what responsibility is (Vincent, 2011) and how they are connected (Bovens et al., 1998; Duff, 2007).

Attributing responsibility to an entity can be both descriptive (e.g., causal responsibility) and normative (e.g., blameworthiness). For the former, one might ask if an agent is responsible for an action or state-of-affairs, while the latter concerns whether one should attribute responsibility to an agent. Responsibility can also be divided into backward-looking notions if they evaluate a past action and possibly lead to reactive attitudes (Wallace, 1994), or forward-looking notions if they prescribe obligations.

Responsibility can take many forms. It not only addresses the moral dimension of society but also tackles legal concepts and other descriptive notions. One can be held legally responsible (i.e., liable) regardless of their moral responsibility, as in the case of strict or vicarious liability. Stating that an agent is causally responsible for a state-of-affairs does not necessarily prescribe a moral evaluation of the action.

Holding an agent “responsible” fulfills a wide range of social and legal functions. Legal scholars state that punishment (which could be seen as a form of holding an agent responsible, e.g., under criminal liability) aims to reform the wrongdoers, deter re-offenses and similar actions, and resolve retributive sentiments (Asaro, 2011; Twardawski et al., 2020). Previous work has addressed how and why people assign responsibility to various agents. The general public might choose to hold a wrongdoer responsible for restoring moral coherence (Clark et al., 2015) or reaffirming a communal moral values (Wenzel and Thielmann, 2006). Psychological research indicates that people base much of their responsibility attribution on retributive sentiments rather than deterrence (Carlsmith et al., 2002), while overestimating utilitarian goals in their ascription of punishment (i.e., responsibility) (Carlsmith, 2008). Intentionality also determines how much responsibility is assigned to an entity (Monroe and Malle, 2017); people look for an intentional agent to hold responsible and infer other entities’ intentionality upon failure to find one (Gray et al., 2014).

AI systems and robots are being widely adopted across society. Algorithms are used to choose which candidate is most fit for a job position (Zhu et al., 2018), decide which defendants are granted bail (Dressel and Farid, 2018), guide health-related decision (Obermeyer et al., 2019), and assess credit risk (Huang et al., 2004). AI systems are often embedded into robots or machines, such as autonomous vehicles (Bonnefon et al., 2016) and robot soldiers (Asaro, 2006). A natural question here is: if an AI system or a robot causes harm, who should be held responsible for their actions and consequences?

In answering this question, some scholars have defended the existence of a (techno-)responsibility gap (Matthias, 2004) for autonomous and self-learning systems.¹¹1Scholars also raise doubt on the existence of techno-responsibility gaps, arguing that moral institutions are dynamic and flexible and can deal with these new technological artifacts (Tigard, 2020b; Köhler et al., 2017). The autonomous component of AI and robots challenges the control condition of responsibility attribution. Simultaneously, their self-learning capabilities and opacity do not allow users, designers, and manufacturers to foresee consequences. Similarly to the “problem of many hands” in the assignment of responsibility to collective agents (Van de Poel, 2015), AI and robots suffer from the “problem of many things,” i.e., current systems are composed of various interacting entities and technologies, making the search for a responsible entity harder (Coeckelbergh, 2019). Scholars have extensively discussed the assignment of responsibility for autonomous machines’ actions and have expanded this gap to more specific notions of responsibility (Asaro, 2016; Koops et al., 2010; Beck, 2016) and its functions (Danaher, 2016; Lima et al., 2020).

Although a clear separation is fuzzy, one may find two schools of thought on the responsibility gap issue. One side argues that designers and manufacturers should take responsibility for any harm caused by their “tools.” (Dignum, 2017; Bryson et al., 2017) Supervisors and users of these systems should also take responsibility for their deployment, particularly in consequential environments like the military as argued by Champagne and Tonkens (Champagne and Tonkens, 2015). The exercise of agency by these systems can be viewed as a human-robot collaboration, in which humans supervise and manage the agency of AI and robots (Nyholm, 2018). Humans should focus on their relationship to the patients of their responsibility to answer for the actions of autonomous systems (Coeckelbergh, 2019). Likewise, other authors argue that society should hold humans responsible because doing so for a machine would be meaningless as it does not understand the consequences of their actions or the reactive attitudes towards them (Sparrow, 2007; Torrance, 2008), possibly undermining the definition of responsibility (Hakli and Mäkelä, 2016).

On the opposite side, some scholars propose autonomous systems could be held responsible per se (Lima and Cha, 2020). From a legal perspective, non-human entities (e.g., corporations) can be held responsible for any damage that they may cause (van Genderen, 2018). These scholars often view these human-AI collaborations as extended agencies where all entities should be held jointly responsible (Hanson, 2009; Gunkel, 2017). AI and robots are part of the socio-technological ensemble, in which responsibility can be distributed across multiple entities with varying degrees (Dodig-Crnkovic and Persson, 2008). These proposals arguably contribute to legal coherence (Turner, 2018), although it could also lead to various repercussions in moral and legal institutions (Beck, 2016). Empirical findings indicate that people attribute responsibility to these systems (Awad et al., 2020; Lima et al., 2020), although to a lesser extent than human agents. According to some scholars, holding AI and robots responsible per se could fulfill specific social goals (Coeckelbergh, 2009) and promote critical social functions (Björnsson and Persson, 2012; Stahl, 2006).

The regulation of AI and robots poses new challenges to policymaking, as in the previously introduced techno-responsibility gap, which society must discuss at large (Coeckelbergh, 2019). The “algorithmic social contract” requires inputs from various stakeholders, whose opinion should be weighed for the holistic crafting of regulations (Rahwan, 2018). It is crucial to understand how people perceive these systems before their wide deployment (Richards and Smart, 2016). Our responsibility practices depend on folk-psychology (Brożek and Jakubiec, 2017) (i.e., how people perceive the agents involved in social practices (Stahl, 2006)). Literature exists on the public perception of moral and legal issues concerning AI (Awad et al., 2020; Lima et al., 2020; Awad et al., 2018). However, little data-driven research has collected public opinion on how responsibility should be attributed for AI and robots’ actions.

A growing number of HCI research has been devoted to understanding how people perceive algorithmic decisions and their consequences in society. For instance, Lee et al. studied people’s perception of trust, fairness, and justice in the context of algorithmic decision-making (Lee, 2018; Lee et al., 2019a) and proposed how to embed these views into a policymaking framework (Lee et al., 2019b). Other scholars explored people’s perceptions of procedural (Grgic-Hlaca et al., 2018) and distributive (Saxena et al., 2019; Srivastava et al., 2019) aspects of algorithmic fairness and studied how they relate to individual differences (Wang et al., 2020; Pierson, 2017; Grgić-Hlača et al., 2020). Nonetheless, little attention is paid to the public attribution of (moral) responsibility to stakeholders (e.g.,  (Lee, 2018; Grgić-Hlača et al., 2019; Robert et al., 2020)), particularly the prospect of responsibility ascription to the AI system per se. The current study contributes by addressing the public perception of algorithmic decision-making through the lens of moral responsibility.

Existing studies addressing how users might attribute blame to automated agents have mostly focused on robots. For instance, Malle et al. observed that people’s moral judgments between human and robotic agents differed in that respondents blamed robots to a more considerable extent had they not taken a utilitarian action (Malle et al., 2015). Furlough et al. found that respondents attributed similar levels of blame to robotic agents and humans when robots were described as autonomous and at the same time the leading cause of harm (Furlough et al., 2019). However, these studies and many others (Li et al., 2016; Wächter and Lindner, 2018; Kim and Hinds, 2006) tackle a singular notion of responsibility related to blameworthiness (Van de Poel, 2015). The present research explores multiple notions of moral responsibility of both human and AI agents involved in decision-making.

AI-based algorithms are now used to assist humans in various scenarios, including high-stakes tasks such as medical diagnostics (Esteva et al., 2017) and bail decisions (Angwin et al., 2016). These algorithms do not make decisions themselves, but rather “advise” humans in their decision-making processes. One such algorithm is the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) tool, used by the judicial system in the US to assist bail decisions and sentencing (Angwin et al., 2016). Several studies have analyzed the fairness and bias aspects of this risk assessment algorithm, e.g., (Dressel and Farid, 2018; Berk et al., 2018; Grgić-Hlača et al., 2019).

This study makes use of publicly available COMPAS data released by ProPublica (Angwin et al., 2016) and considers the machine judgments as either an AI advisor (later in Study 1) or an AI decision-maker (in Study 2). As stimulus material, we use real-world data obtained from a previous analysis of the tool (Angwin et al., 2016), which focused on its application in bail decision-making. This dataset contains information about 7,214 defendants subjected to COMPAS screening in Broward County, Florida, between 2013 and 2014.

We use 100 randomly selected cases from this dataset, the corresponding bail suggestions, and information about whether the defendant re-offended within two years of sentencing. The sampled data was balanced concerning these variables. Each defendant’s COMPAS score ranges from 1 to 10, with ten indicating the highest risk of re-offense or nonappearance in court. In this study, scores 1 to 5 were labeled “grant bail” and 6 to 10 were labeled “deny bail.”

Ascribing responsibility is a complex moral and legal practice that encompasses various functions, entities, and social practices (Mulligan, 2017; Stahl, 2006). Responsibility has multiple distinct meanings depending on its purpose and requirements. The current study revisits eight notions of responsibility compiled from psychology and philosophy. All of these notions originated from Van de Poel’s work (Van de Poel, 2011, 2015), except for responsibility-as-authority and as-power, which comes from Davis’s discussion on professional responsibility (Davis, 2012). We complement these notions with a wide range of literature ranging from philosophical theories of moral responsibility (e.g., (Shoemaker, 2011; Scanlon, 2008)) to approaches in the context of AI systems (e.g., (Coeckelbergh, 2019; Torrance, 2008)). Although not exhaustive (e.g., we have not addressed virtue-based notions of responsibility as they cannot be easily adapted to AI systems), we highlight how our work differs from previous HCI approaches.

• •

  Responsibility-as-obligation:
  E.g., “The (agent) should ensure that the rights of the defendant are protected.”
  One could be held responsible-as-obligation through consequentialist, deontological, and virtue-based routes (Van de Poel, 2015). While an entity could be attributed such meaning of responsibility based on pre-determined consequentialist distribution principles, the latter two routes presuppose the agent’s initiative or promise to see to it that a specific state-of-affairs is brought about. This notion differs from responsibility-as-task as it does not imply that one should be the agent to bring about a specific state-of-affairs, but rather indicates that one should fulfill its supervisory duties in the process.

• •

  Responsibility-as-task:
  “It is the (agent)’s task to protect the defendant’s rights.”
  This descriptive notion of responsibility ascribes a specific task to an entity. These assignments do not necessarily define a moral obligation per se (Van de Poel, 2011) and are often accompanied by the understanding that an entity has to do something by itself (Davis, 2012). In our experimental design, we highlight the agent’s acting role in completing its task.

• •

  Responsibility-as-authority:
  “The (agent) has the authority to prevent further offenses.”
  To be responsible-as-authority implies that one is in charge of a specific action or state-of-affairs. This notion has also been posed as ”responsibility-as-office” by Davis (Davis, 2012) in the context of engineers’ professional responsibility. An important aspect of responsibility-as-authority is the possibility of delegating other complementing notions, such as responsibility-as-task, to other agents. We address this meaning of responsibility by explicitly indicating that the agent has the authority in bail decisions.

• •

  Responsibility-as-power:
  “The (agent) has the skills needed to protect the rights of the defendant.”
  If an entity has the skills needed to bring about an action or state-of-affairs, one might ascribe it responsibility-as-power (Davis, 2012). In other words, having the ability, in terms of competency, knowledge, or expertise, might lead to the assignment of this notion of responsibility.

• •

  Responsibility-as-answerability:
  “The (agent) should justify their advice.”
  This is related to how one’s reasons for acting in a specific manner could be seen under moral scrutiny. Shoemaker proposed this notion of moral responsibility as a form of judgment of one’s actions grounded in moral evaluations (Shoemaker, 2011). Davis proposed a similar meaning of responsibility under a different name, responsibility-as-accountability (Davis, 2012), as the responsibility for explaining specific consequences. Coeckelbergh later applied this concept through a relational approach for actions and decisions made using AI (Coeckelbergh, 2019).

• •

  Responsibility-as-cause:
  “The (agent)’s decision led to the prevention of the re-offense.”
  This meaning of responsibility has been further discussed depending on the nature of an action’s consequences (Davis, 2012), e.g., being causally responsible for a positive state-of-affairs could lead to the ascription of “good-causation.” Causality is also an important pre-condition for other normative notions of responsibility, such as blame, as the blurring of a causal connection raises questions on the foreseeability and control of a specific action. (Van de Poel, 2015; Malle et al., 2014)

• •

  Responsibility-as-blame/praise:
  “The (agent) should be blamed for the violation of the rights of the defendant.” / “The (agent) should be praised for the protection of the rights of the defendant.”
  Blaming an entity for the consequences of their actions has been debated as adopting certain reactive attitudes towards it (Wallace, 1994). Scholars have also argued that to blame someone is to respond to “the impairment of a relationship,” (Chislenko, 2019; Scanlon, 2008) especially towards its constitutive standards (Shoemaker, 2011). Scholars have debated the possibility of ascribing blame to an automated agent and agree that doing so would not be morally appropriate (Danaher, 2016; Torrance, 2008). Regardless of this consensus, previous studies have found that people attribute a similar degree of blame to robotic and human agents under specific conditions (e.g.,  (Malle et al., 2015; Furlough et al., 2019)).
  As an opposite concept of blame, one may consider “praise” as a positive behavioral reinforcement (Kazdin, 1978) through which one conveys its values and expectations of the agent (Delin and Baumeister, 1994). Hence, we consider both blame and praise as responsibility notions in this research.

• •

  Responsibility-as-liability:
  “The (agent) should compensate those harmed by the re-offense.”
  An entity that is ascribed this responsibility should remedy any harm caused by their actions (Van de Poel, 2015). Rather than dwelling on the discussion addressing the mental states of AI and robots and their arguable incompatibility with criminal law and its assumption of mens rea (Lima, 2017; Lagioia and Sartor, 2019; Gless et al., 2016), we address this notion from a civil law perspective. Scholars propose ‘making victims whole’ as the primary goal of tort law (Prosser et al., 1941), and hence, we similarly address responsibility-as-liability. We also add that the idea of holding automated agents liable became prominent after the European Parliament considered adopting a specific legal status for “sophisticated autonomous robots” (Delvaux, 2017). Nevertheless, it is important to note that current AI systems cannot compensate those harmed, as they do not possess any assets to be confiscated (Bryson et al., 2017).

Imagine that you read the following story in your local newspaper: A court in Broward County, Florida, is starting to use an artificial intelligence (AI) program to help them decide if a defendant can be released on bail before trial. Early career judges are taking turns receiving advice from this AI program and another human judge, hired to serve as an advisor.

We employed a factorial survey design (Wallander, 2009) and showed participants eight vignettes that described a defendant from the ProPublica dataset, information about who the advisor was (i.e., an AI program or a human judge), which advice they gave, what the judge’s final decision was, and whether the defendant committed a new crime within the next two years (i.e., re-offended). All vignettes stated that the judges’ final decision followed the advice given, given the ProPublica dataset does not provide this information. After reading the stimulus material, respondents were asked to indicate to what extent they agreed with a set of statements, presented in random order between participants, regarding the advisor on a 7-point Likert scale (-3 = Strongly Disagree, 3 = Strongly Agree).²²2 Questions related to responsibility-as-liability were shown in scenarios where i) the defendant re-offended and the phrasing style addressed the prevention of re-offenses, or ii) the defendants were denied bail and did not re-offend within two years while the statements focused on protecting their rights. The phrases tackling praise and blame were presented depending on the advice/decision and recidivism. These statements aimed to capture different notions of responsibility (see Table 2 in the Appendix for the complete list). Figure 1 illustrates the survey methodology. Participants were also asked two attention check questions in between vignettes.

Each participant in the study was exposed to a random subset of four cases with human advice and another four with AI advice. We ensured a balanced set was shown to each participant in terms of the advice (i.e., grant bail vs. deny bail) and recidivism. As a result, each respondent was shown one vignette of every possible combination of scenarios, encompassing eight (advice $\times$ recidivism $\times$ AI vs. human) variations. All vignettes were presented in random order to eliminate any order effect (Redmiles et al., 2017; Groves et al., 2011).

Bail decisions aim to procure a balance between protecting future victims, e.g., prevent further offenses, and to impede any unnecessary burdens towards the defendant, e.g., by ensuring that their rights are protected (Grgić-Hlača et al., 2019). The latter aspect of bail decisions is related to the assumption that one is innocent until proven otherwise beyond a reasonable doubt under criminal law (Dhami et al., 2015). To strike a balance between these two functions of bail decisions, we phrase statements addressing all notions of responsibility addressed in this work in two different forms: a human agent or an AI program could be held responsible for i) (not) protecting the rights of the defendant and ii) (not) preventing re-offense. Participants were randomly assigned to one of these treatment groups, and all statements followed the same phrasing style.

Towards the end of the survey, we asked demographic questions (presented in Table 1). We also gathered responses to a modified questionnaire of NARS (Negative Attitude towards Robot Scale) (Syrdal et al., 2009), whose subscale addressed “artificial intelligence programs” rather than “robots” to accommodate the COMPAS tool.

We interviewed six demographically diverse respondents. Three respondents were recruited through the online crowdsourcing platform Prolific (Palan and Schitter, 2018), while the other three were our colleagues, who had prior experience designing and conducting human-subject studies. After each interview, we iteratively refined our survey instrument based on the respondent’s feedback. We stopped gathering new responses once the feedback stopped leading to new insights. This process led to two significant changes in our survey instrument design. Firstly, we adapted the vignette presentation, which was initially adapted from previous work (Dressel and Farid, 2018). Our respondents unanimously stated that they found information about defendants easier to read, understand, and use when presented in a tabular format (shown in Figure 4 in the Appendix). Secondly, we rephrased some of the statements about the notions of responsibility we address in this work so that survey respondents’ understanding of these concepts is similar to the definitions introduced above.

We conducted a power analysis to calculate the minimum sample size. A Wilcoxon-Mann-Whitney two-tailed test, with a 0.8 power to detect an effect size of 0.5 at the significance level of 0.05, requires 67 respondents per treatment group. Hence, we recruited 400 respondents through the Prolific crowdsourcing platform (Palan and Schitter, 2018) to compensate for attention-check failures. We targeted US residents who have previously completed at least 100 tasks on Prolific, with an approval rate of 95% or above. Each participant was randomly assigned to one of the two studies.

The respondents’ demographics are shown in Table 1. Prior studies of online crowdsourcing platforms have found that respondent samples tend to be younger, more educated, and consist of more women than the general US population (Ipeirotis, 2010). Compared to the 2016 US census (U.S. Census Bureau, 2016), our respondents are indeed younger and more highly educated. However, both of our studies’ samples have a smaller ratio of women than the US population. Asian ethnicity is slightly over-represented in our samples. Compared to Pew Research data on the US population’s political leaning (Pew Research Center, 2016), our samples are substantially more liberal.

The respondents were remunerated US$\$10.5$ for taking part in the cognitive interviews and US$\$1.66$ for completing the online surveys. The cognitive interviews lasted less than 30 minutes, while the online surveys took 10.36$\pm$5.43 minutes. Hence, all study participants were paid above the US minimum wage.

Figure 2(b) shows how people attributed each notion of responsibility to AI and human agents in Study 1 (on the advisor role) and Study 2 (on the decision-maker role).

First, responsibility-as-answerability (i.e., the bar in the middle) was the notion ascribed the highest to both human and AI advisors and decision-makers, followed by responsibility-as-obligation, as-task, as-authority, and as-power (i.e., the first four bars). On the other hand, liability and blame were the least attributed responsibility notion in bail decisions. Responsibility-as-cause and praise were the most neutral notions, and their mean attribution is close to zero (i.e., the baseline) across all treatments (see Figure 5 in the Appendix).

Second, Figure 2(b) shows two distinct sets of responsibility notions, where these clusters can be observed from the pairwise Spearman’s correlation chart. A high correlation value indicates that those responsibility notions are perceived similarly by people. One group includes responsibility-as-task, as-authority, as-power, and as-obligation, all of which have positive mean values. The other group includes responsibility-as-cause, praise, blame, and liability. Responsibility-as-answerability belongs to neither of these groups.

Third, we can quantify variations across vignette conditions. Each vignette shown to participants varied in the advice given, bail decision, and recidivism, allowing us to compare across these factors. Our data show that vignettes that grant bail (as opposed to denying bail) led to a higher assignment of all responsibility notions, particularly causal responsibility and blame (see Figure 5 in the Appendix). A similar effect was found depending on defendant recidivism. For instance, the first four responsibility notions were ascribed to a more considerable degree if the defendant did not re-offend. In contrast, responsibility-as-cause, blame, and liability were attributed to a lesser extent if the defendant re-offended within two years. These trends corroborate the responsibility clusters discussed above.

Finally, our study participants were also assigned to one of two different phrasing styles addressing some of the bailing decisions’ objectives. Except for responsibility-as-answerability, addressing the violation or protection of a defendant’s rights led to a marginally higher assignment of responsibility than the phrasing style focusing on preventing re-offenses.

Our primary goal was to examine how people attribute responsibility to human and AI agents in high-stakes scenarios. To quantify the difference, we used a multivariate linear mixed model that included a random-effects term to control for each participant. This allowed us to account for repeated measures, i.e., explicitly model that each participant responded to questions on eight distinct defendants. We use the standard .05 level of significance. In all models, we use our adapted scale of pre-attitude towards AI systems as a control variable. Figure 3(b) shows the results. The annotated numbers indicate the differences and significance levels between the two agents. We report the full regression coefficients in Table 3 in the Appendix.

Both Study 1 and Study 2 show consistent differences in responsibility attribution between agents, regardless of whether they informed a human judge (Study 1) or decided by themselves (Study 2). We note subtle differences in how people attribute responsibility to AI and humans. The first four responsibility concepts are correlated; the notions addressing tasks, supervisory roles, and the skills needed to assume them show a meaningful difference between agent types. The respondents attributed more of these notions of responsibility to humans than to AIs.

Responsibility-as-answerability exhibits a marginal difference with respect to the agent type that assisted human judges in bail decisions; however, the same trend was not observed in Study 2. Nevertheless, our results suggest that humans and AI are judged similarly responsible with respect to causality, blame, and liability for bail decisions. Moreover, human decision-makers are praised to a considerably larger degree than AI decision-makers, although the same effect was not observed for human and AI advisors.

So far, we have observed two clusters of responsibility concepts by their correlation. The first cluster is composed of responsibility-as-task, authority, power, and obligation — all of which were attributed to a greater degree to humans than AI systems ($\Delta$>0.206, $p$<.001). The first three are descriptive and focus on one’s tasks (i.e., task, authority) and the necessary skills for their completion (i.e., power). Furthermore, responsibility-as-obligation is related to responsibility-as-task in prescribing a specific goal to the agent; it differs from the latter, however, in setting a supervisory role towards the task, rather than specifying that one should be the one to complete it.

The second cluster includes causal responsibility, blame, praise, and liability — all of which were attributed to a similar degree to humans and AI. This finding is in line with previous work on blame assignment, highlighting the significance of causality in people’s ascription of blame and punishment. Human subject studies suggest that blame attribution is a two-step process; it is initiated by a causal connection between an agent’s action and its consequences and is followed by evaluating its mental states, i.e., intentions (Cushman, 2008). Malle et al. (Malle et al., 2014) have also proposed a theory of blame that is dependent on the causal connection between an agent and a norm-violating event. Our data similarly reveal such a relationship, even when controlling for the advice given, bail decision, or re-offense.

Concerning the phrasing styles, our experiment design addressed responsibility-as-liability as the duty to compensate those harmed by a wrongful action. However, previous work on the connection between liability (i.e., punishment) and causality focuses on the retributive aspect of punishment (Cushman, 2008), often drawing a connection between punishment and blame. Therefore, we do not posit that people’s ascription of liability is solely dependent on causality determinations. We hypothesize that the low assignment of liability is due to the current study’s bail decision-making context. For instance, those wrongfully convicted do not receive any compensation for years spent in prison in at least 21 US states (Slifer, 2014). Hence, people might not believe that compensation is needed or deserved, or attribute this notion of responsibility to other entities, such as the court or the government, leading to a lower ascription of liability to the advisor or decision-maker.

Our findings indicate that participants who were presented with responsibility statements addressing the violation or protection of a defendant’s rights (e.g., “It is the AI program’s task to protect the rights of the defendant”) were assigned higher responsibility levels across all notions. We posit that this effect results from the control that judges (humans and AIs) have over the consequences of their advice and decisions. Although a judge’s decision can directly affect a defendant’s rights depending on the appropriateness of one’s jailing, preventing re-offenses is a complex task that encompasses diverse factors, such as policing and the defendant’s decision to re-offend.

Participants perceived human judges and advisors as more responsible for their tasks than their AI counterparts (see the leftmost bars in Figure 3(b)). Humans are responsible for the tasks they are assigned, e.g., preventing re-offenses because they are in charge (i.e., authority) and have the skills necessary for completing them (i.e., power). These agents should either oversee (i.e., obligation) these tasks or take the lead (i.e., task). On the other hand, AI systems are ascribed lower levels of all these responsibility notions.

The meanings of responsibility addressing the attribution of tasks and their requirements are descriptive in the sense that they should be addressed in the present tense (Davis, 2012), e.g., one is responsible for a task, or is in charge of it. Although descriptive and present-looking, these notions lead to the prescription of forward-looking responsibilities, such as an obligation. For instance, to be responsible for a specific task because one has the authority and necessary skills prescribes that one should see to it that the task is completed, i.e., an obligation is prescribed, through consequentialist, deontological, or virtue-based routes (Van de Poel, 2015).

Participants attributed lower levels of authority and power to AI. This indicates that these systems are not thought to possess the necessary abilities to make decisions and advise such high-stakes decisions. Therefore, it is not deemed the AI program’s responsibility to complete the assigned task or see it to be fulfilled.

One of the prominent findings of this work is the need of interpretable AI systems. Although our participants assign a marginally lower level of responsibility-as-answerability for AI advisors vis-à-vis their human counterparts ($\Delta$=0.167, $p$<.05), they believe they should justify their decisions to the same extent as human judges, particularly if they are to make the final bail decision ($p$>.05).

Moreover, our results suggest that an AI without a human-in-the-loop, i.e., AI judges in Study 2, could be held at the same level of scrutiny as human decision-makers for their decisions. This finding may imply that deploying black box AI in high-stakes scenarios, such as bail decision-making, will not be perceived well by the public. There exists empirical evidence that people might be averse to machines making moral decisions (Bigman and Gray, 2018). Previous work has not controlled for a system’s interpretability, and therefore such trends might either i) be caused by the lack of explanations or ii) be aggravated if people become aware that AI systems cannot justify their moral decisions.

Judges should base their decisions on facts and be able to explain why they made such decisions. AI systems should also be capable of justifying their advice and decision-making process according to our results. This finding demonstrates the significance of these systems’ interpretability. Scholars have discussed the risks posed by the opacity of existing AI algorithms. They argue that understanding how these systems come to their conclusions is necessary for both safe deployment and wide adoption (Floridi et al., 2018). Explainable AI (XAI) (Gunning, 2017) is a field of computer science that has been given much attention in the community (Gilpin et al., 2018), and our results suggest that people agree with its importance.

Previous work has found that one’s normative and epistemological values influence how explanations are comprehended (Lombrozo, 2009). Explanations involve both an explainer and explainee, meaning that conflicts might arise concerning how they are evaluated (Mittelstadt, 2019). Therefore, we also posit that future work should delve deeper into what types of explanations the general public expects from AI systems. We highlight that those who are in charge of developing interpretable systems should not try to “nudge” recipients so they can be manipulated (Lipton, 2018), e.g., for agency laundering (Rubel et al., 2019).

The four rightmost bars in Figure 3(b) suggest that AI and human agents are ascribed similar levels of backward-notions of responsibility, namely blame, liability, praise, and causal responsibility.