Explainability and AI Confidence in Clinical Decision Support Systems: Effects on Trust, Diagnostic Performance, and Cognitive Load in Breast Cancer Care

Numerous studies in the literature have explored various aspects of technology adoption, leading to the development of models such as the Technology Acceptance Model (TAM) [13], widely applied in healthcare studies [19], Task-Technology fit (TTF) [20] and the Unified Theory of Acceptance and Use of Technology (UTAUT) [21]. TAM, in particular, offers a foundational framework for understanding users’ adoption of technology by identifying two primary constructs, Perceived Usefulness (PU) and Perceived Ease of Use (PEOU). While TAM addresses the cognitive aspects of technology adoption, trust extends it by capturing emotional and psychological dimensions [22], making it a critical factor in overcoming situations involving uncertainty, dependency, and risks and enhancing acceptance [23, 24, 12, 25].

Trust has been defined in various ways by scholars [26, 27, 28], with a widely accepted definition describing it as an individual’s willingness to take risks and delegate critical tasks and responsibilities to another party, such as a technological system [26]. In healthcare, however, clinicians often hesitate to rely on automated systems, choosing instead to depend on their own expertise. This reluctance creates significant barriers to the acceptance and adoption of advanced technologies. Considering trust as a critical factor in technology acceptance [6, 14, 15], many studies have focused on understanding how it can be built between users and technological systems.

Some frameworks, such as Muir’s trust model [28] and Mayer et al.’s influential model focusing on ability, benevolence, and integrity [27], provide foundational insights into trust. In the context of human-technology interactions, Parasuraman et al. [11] expanded the understanding of trust by examining user reliance on automation and its potential for misuse or neglect. They highlighted the risks of ”automation abuse,” where over-automation can undermine human oversight, leading to errors or reduced performance. Studies such as [10] highlight the risks associated with overreliance on automation, including reduced monitoring of systems and the potential for biased decisions. These findings highlight the importance of designing systems that emphasize on calibrated trust to improve the humans’ task performance [10, 26, 29, 30, 31, 32].

Trust formation in human-technology interactions has been widely explored, with Hoff and Bashir [33] categorizing trust development into dispositional trust, situational trust, and learned trust. Building on these categories, other researchers have proposed alternative frameworks. For instance, Jermutus et al. [34] classify trust factors into user-related, AI-related, and contextual factors, while Lotfalian et al. [35] identify three key categories: human user attributes, intelligent system design, and task characteristics. Schaefer et al. [36] mentioned the factors differently as automated partner, the environment in which the task is occurring, and characteristics of the human interaction partner.

AI-related factors, such as explainability [37], transparency [25], and complexity [38], are fundamentals to enhancing user trust and reliance on AI systems. Improvements in these areas can directly address user uncertainties, ensuring outputs are clear, interpretable, and reliable. By enhancing these capabilities, AI systems empower users to make informed decisions with greater confidence, supporting appropriate reliance and facilitating broader system adoption. Explainability has been recognized as a key factor in improving trust and reliance on AI systems [15, 39, 40, 31, 41, 37]. Users need to understand how the system generates its decisions and the reasoning behind the algorithms [34].

Explainable Artificial Intelligence (XAI) has become essential in fields like medicine[42, 31], finance, and autonomous systems, where understanding AI decisions is critical. As complex “black-box” models like deep learning grow in use, their lack of transparency raises concerns about trust and accountability. XAI aims to address this by providing clear, human-understandable explanations, making AI systems more transparent and reliable [43]. This not only builds user confidence but also supports effective human-AI collaboration [43, 17, 37, 34].

In the literature, various explainability methods have been introduced to address the black-box nature of AI models, which are often categorized into local and global approaches. Local methods, such as LIME (Local Interpretable Model-agnostic Explanations) [44], Saliency Maps [45], Grad-CAM (Gradient-weighted Class Activation Mapping) [46], Integrated Gradients [47],and example-based methods, such as counterfactual explanations [48] and prototype-based techniques [49], focus on explaining individual predictions by identifying specific features that influenced a particular output. In contrast, global methods, like SHAP (SHapley Additive Explanations) [50], Feature Importance, and Partial Dependence Plots (PDPs) [51], provide insights into the overall behavior of the model, showing how features contribute to predictions across the entire dataset.

Several studies in the literature have explored the impact of explainability methods on participants’ behavior, particularly focusing on trust and adoption. Some studies have demonstrated positive effects, showing that explainability enhances trust and supports adoption. For example, Wang et al. [43] found that using SHAP improved decision accuracy and behavioral trust in sales prediction tasks. similarly, Leichtmann et al. [52] conducted a 2×2 between-subject study on a mushroom-picking task, using Grad-CAM as an attribution-based explanation and ExMatchina as an example-based explanation. Both methods were found to significantly improve trust in AI systems, highlighting their effectiveness in enhancing user confidence.

However, other studies have reported limited or no significant effects of explainability on trust. For instance, in our previous study [53], no improvement was observed when providing different levels of explainability, such as AI confidence scores, localization, and other methods. Ahn et al. [54] found that while SHAP and LIME increased interpretability, they did not significantly enhance trust or task performance, emphasizing the role of outcome feedback over explainability. Similarly, Cecil [41] observed that explainability methods such as saliency maps and visual charts had minimal impact on mitigating the negative effects of incorrect AI advice in human resource management. Zhang et al. [17], also, used SHAP as a local explainability method in an income prediction task but found no significant impact on trust or user behavior.

Additionally, research comparing the effectiveness of different XAI methods has shown mixed results, with some methods being more impactful than others. For instance, Alam et al. [55] reported that richer explanations, such as a combination of written, visual, and example-based methods, were more effective in improving satisfaction and trust in AI diagnosis systems compared to simpler approaches. Similarly, Cai et al. [56] explored both normative and comparative example-based explanations in a sketch recognition task. Their findings revealed that normative explanations positively impacted system comprehension and user trust, while comparative explanations did not show significant effects. Wang et al. [57] also evaluated XAI methods in recidivism and forest cover prediction tasks, comparing global approaches like feature importance with local methods such as counterfactual explanations and nearest-neighbor examples. The study found that the effectiveness of XAI largely depends on users’ domain expertise, providing insights for tailoring XAI to support decision-making. The study most closely aligned with the goals of our research is by Evans et al. [58], which investigates how digital pathologists interact with various XAI tools, including saliency maps, concept attributions, prototypes, trust scores, and counterfactuals, to enhance clarity in medical imaging recommendations. While their study provides valuable insights into pathologists’ preferences for AI guidance, it does not examine how these tools and varying levels of explainability influence human-related behaviors, such as trust, reliance, cognitive load, and decision-making, which is a central focus of our work.

Several studies have investigated AI confidence scores as an explainability feature to build trust. Zhang et al. [17] found that confidence scores help calibrate trust by providing insights into the certainty of predictions. McGuirl and Sarter [59] showed that continually updated confidence information improves trust calibration in automated decision support systems. Similarly, Antifakos et al. [18] emphasized the importance of how confidence is presented in an automatic notification device. Rechkemmer and Yin [60] further demonstrated that confidence scores significantly influence trust, although accuracy remains a stronger factor.

Cognitive load refers to the mental effort required to process information and complete tasks [61]. Cognitive Overload can negatively impact decision-making and reduce trust in automated systems, as users may feel overwhelmed or struggle to understand the system’s outputs [62]..

Research has explored the relationship between cognitive load and trust across various domains. For instance, Choudhury et al. [61] found that increased cognitive workload reduced the intent to use a blood utilization calculator (BUC), an AI-based decision support system in healthcare. This underscores the importance of designing AI systems that minimize cognitive load to foster trust and encourage adoption. However, not all studies find a direct link between explainability and cognitive load; for example, Wang et al. [63] reported no significant differences in cognitive load between groups provided with XAI explanations versus AI-only systems.

In the context of explainable AI, the effect of explainability on cognitive load varies depending on the context and implementation. Few studies provide evidence of this variability. For instance, Herm [64] conducted an empirical study in medicine, using chest X-ray images of COVID-19-infected patients, and demonstrated how different XAI methods influence cognitive load. Similarly, Kaufman et al. [65] explored cognitive load in the context of autonomous driving, and Hudon et al. [66] examined the effects of XAI in image classification tasks. These findings highlight the critical need to consider human cognitive factors when designing XAI systems to ensure explanations are intuitive and do not inadvertently increase cognitive load. By carefully balancing transparency and usability, XAI can enhance trust without overwhelming users.

This experimental study addresses research questions using online experiments with multiple intervention conditions. Each condition was designed to include different levels of explainability, to evaluate clinicians’ attitudes toward an AI-based Clinical Decision Support System (CDSS) for breast cancer diagnosis. During the experiment, participants interacted with the CDSS across various diagnostic cases, assessing their trust and agreement with the system’s suggestions. In addition, participants completed surveys provided demographic information before the experiment and evaluated their mental demands and stress after using each CDSS variant.

The following sections provide a detailed description of the study participants, experimental design and procedures, the specific AI configurations employed across intervention conditions, measurements, and the analytical methods used.

A total of 28 participants were recruited between January and August 2024 through medical associations, social platforms, and professional networks. All participants were U.S.-based, fluent in English, and over the age of 18. They were compensated $80 for their participation. The sample comprised primarily radiologists ($\approx 60\%$), with the remaining participants identified as oncologists ($\approx 18\%$) or other healthcare professionals involved in breast cancer care ($\approx 22\%$).

The demographic characteristics of the participants were as follows: The majority were male ($\approx 60\%$), and the mean age was 42.6 years ($SD=12.1$), with most participants ($\approx 53\%$) in the 35-45 age. In terms of race and ethnicity, the majority identified as White ($\approx 68\%$), followed by Asian ($\approx 14\%$), Middle Eastern or North African ($\approx 7\%$), Hispanic or Latino ($\approx 3\%$), and other backgrounds ($\approx 7\%$).

The primary goal of this study was to evaluate the effect of different levels of explainability on participants’ cognitive load. We used an interrupted time series design [67], where participants sequentially interacted with a breast cancer Clinical Decision Support System (CDSS) across multiple interventions, each offering a distinct level of explainability. Participants completed these interventions in a particular order, with key variables measured at multiple points to capture changes over time.

The main task required participants to diagnose ultrasound images of breast tissues as either healthy, benign, or malignant. Initially, they made these diagnoses without any decision support. As they progressed through the interventions, they were gradually provided with diagnostic suggestions from the CDSS, with varying levels of explanation generated through machine learning techniques. The experiment included pre-experiment surveys to collect demographic information and baseline attitudes toward AI. Each intervention involved diagnosing 10 images, followed by a brief post-intervention survey.

The entire experiment was conducted online on a web application developed using the Python-based Dash framework. The online application platform integrated all stages, from signing the consent form and completing surveys to participating in the experiment sessions and collecting data. This setup allowed us to recruit participants nationwide without requiring in-person supervision.

After recruitment, participants received a video tutorial and an experiment link via email. This approach ensured that only those genuinely interested and informed about the study participated, helping to maintain data authenticity. Upon accessing the link, participants first signed a consent form, completed a pre-experiment survey, and then proceeded through a series of experiment sessions as follows:

• •

  Baseline (Stand-alone): Clinicians review breast cancer tissue images and make diagnostic decisions independently, without any suggestions.

• •

  Intervention I (Classification): Clinicians receive diagnostic recommendations (healthy, benign tumor, malignant tumor) from the system, but without any explanatory details.

• •

  Intervention II (Probability Distribution): Expanding on the previous intervention, this stage includes probability scores for each diagnostic option (healthy, benign tumor, malignant tumor).

• •

  Intervention III (Tumor Localization): In addition to probability estimates, the system highlights the suspected location of the tumor within the breast tissue images.

• •

  Intervention IV (Enhanced Localization with Confidence Levels): Building on the third intervention, clinicians are provided with tumor localization information that includes areas marked with both high and low confidence levels.

In the baseline condition, participants analyzed ultrasound images independently and provided their diagnosis as healthy, benign, or malignant. In the subsequent conditions, participants were given diagnostic suggestions and asked to rate their agreement and trust in these suggestions on a 5-point Likert scale (as shown in Figure 1). If their agreement rating was below 3, they were prompted to make their own diagnosis and share their opinion on the image (see Figure 1).

After completing each intervention, participants proceeded to a post-intervention survey designed to capture specific insights about that stage. Using a 5-point Likert scale, they rated aspects such as cognitive load and other relevant factors. The full procedure for the experiment is outlined in Figure 2.

The AI system supporting the CDSS is based on our previous work [68] and combines a U-Net model for precise image segmentation with a Convolutional Neural Network (CNN) for diagnostic classification. The U-Net architecture [69] is specifically designed to delineate the boundaries of suspicious regions within breast ultrasound images, effectively highlighting areas that may contain cancerous tissue. This segmentation step assists in localizing potential tumors and provides visual cues that clinicians can use to focus their analysis in 3rd and 4th Interventions. Following this step, the segmented images are analyzed by the CNN, a widely used method for analyzing medical images [70, 71, 72], which categorizes each image into one of three diagnostic classes: healthy, benign, or malignant. This classification model was trained using a publicly available breast cancer ultrasound dataset containing 780 images [73]. The training process enabled the model to learn and recognize distinguishing features associated with each diagnostic category, achieving an overall diagnostic accuracy of 81%.

In the pre-experiment survey, we assessed participants’ perceptions of AI through three key measures: AI role, AI usefulness, and complexity perception.

The mixed-effects linear model analyses examined the impact of different levels of AI explainability on clinicians’ mental demand and stress. The models included intervention conditions as fixed effects and participant groups as random effects, controlling for inter-individual variability.

The mixed-effects model was used to assess the influence of AI confidence score (No confidence, low and high) on four key outcomes: trust, agreement, performance, and diagnosis duration. The results are summarized in Table 2.

The ANOVA results presented in Table 3 represent the influence of demographic variables on various outcome measures concerning perceptions and experiences of participants. The outcome measures which were considered in this regard are mental demand, stress, AI role, complexity perception, and AI usefulness. Significant effects are identified based on the F-statistic along with p-values across these variables.

Results reveal that age is a highly significant factor across all outcome measures, with extremely low p-values (all below $10^{-5}$), indicating strong relationships between age groups and their responses. Similarly, experience bracket significantly affects certain outcome measures, namely, mental demand ($p=2.77\times 10^{-4}$), AI role ($p=1.00\times 10^{-6}$), and AI usefulness ($p=3.76\times 10^{-3}$), pointing to a relationship between participants’ experience levels and their perceptions of these specific factors.

Job role also shows a significant effect on perceptions, especially regarding AI role ($p=4.82\times 10^{-4}$), complexity perception ($p=2.58\times 10^{-5}$), and AI usefulness ($p=1.37\times 10^{-2}$). This implies that job responsibilities may shape how participants view AI functionality and complexity. Race has a significant impact on all outcome measures as well, highlighting potential differences in AI-related perceptions across racial backgrounds. Lastly, gender plays a significant role specifically in perceptions of complexity perception ($p=6.61\times 10^{-16}$) and AI usefulness ($p=4.30\times 10^{-6}$), suggesting that men and women might perceive AI complexity and usefulness differently.

When it comes to AI usefulness, female oncologists perceive AI as the most useful, whereas male radiologists show lower ratings, suggesting that gender and job role impact how useful AI is perceived within these fields. Finally, for AI role, radiologists report a generally high perception of AI’s role, with male oncologists rating it lowest, indicating that radiologists may view AI as more integral to their field compared to other job roles. These findings underscore how gender and professional role shape perceptions of stress, mental demand, task complexity, and the role and usefulness of AI, with notable differences seen in the oncologist and radiologist groups.

The box plots illustrate how participants’ perceptions of AI role, complexity perception, AI usefulness, mental demand, and stress vary across different demographic categories: age, gender, and race. For age (Figure 3), younger participants (25-34) tend to rate AI role and AI usefulness lower, while older participants, especially those 55 and above, rate these aspects higher, suggesting that older participants see more value in AI’s role and utility. Notably, older participants report lower mental memand and stress levels, suggesting that their greater experience may reduce the cognitive load required for these tasks compared to younger participants.

Gender differences (Figure 5) reveal that female participants perceive the AI role as less significant compared to male participants but report higher levels of complexity and stress. This suggests that, while females may view AI as having a lesser role, they find the tasks associated with it more complex and demanding than their male counterparts. Racial differences are also notable as shown in Figure 4: middle eastern or north African participants rate AI role, complexity, and AI usefulness highly, indicating a generally positive perception of AI’s potential and relevance. In contrast, participants from the Asian group report higher levels of Stress, suggesting that they may experience more cognitive or emotional strain when engaging with AI-related tasks. These findings suggest that age, gender, and race shape individuals’ views on AI’s utility and complexity, as well as the cognitive and emotional demands associated with interacting with AI systems.

Most studies focus on the effect of explainability on cognitive load by examining individual explanation models [75, 66] or comparing different models [64, 76]. To answer RQ1, this study broadens the scope by exploring the impact of varying levels of AI explainability, from no explanation to increasingly detailed explanations, on cognitive load, specifically mental demand and stress.

Karran et al. [77] argue that any form of XAI explanation can reduce cognitive effort by aligning users’ mental models with the task. Our findings align with this perspective, showing that increasing levels of explainability generally decreased mental demand. However, this reduction was not statistically significant, consistent with Wang et al. [63], who reported no significant differences in cognitive load between groups provided with XAI explanations and those using AI-only systems. Notably, explanations that were concise, visually clear, and focused on relevant information showed the greatest potential for reducing mental load.

In contrast, the fourth condition, which introduced more complex and detailed information, resulted in increased mental demand. This highlights that overly detailed or poorly aligned explanations can conflict with users’ mental models, requiring additional cognitive effort to process. These findings underscore the importance of designing explanations that are both clear and relevant, as their quality significantly affects cognitive load [64]. By prioritizing user-centered explanation design, XAI systems can better support users without introducing unnecessary cognitive strain.

On the other hand, stress levels increased for all levels of explainability, with the third condition, tumor localization with probability estimates, causing a statistically significant rise. This increase can be attributed to the distinct nature of the information introduced in this condition, which required greater cognitive effort to interpret and apply. Consistent with previous studies, such as [75], explanations can impose additional cognitive demands depending on the style and complexity of the information provided.

These findings highlight the need for clear and concise XAI design. Overly complex explanations can increase cognitive demands, while appropriate information helps reduce strain and improve usability [65].

Confidence scores play a crucial role in helping users calibrate their trust in AI systems, enabling them to determine when to trust or distrust AI outputs [17]. The effectiveness of displaying confidence information has been observed across diverse domains, such as air traffic control, military training, and mobile applications, highlighting its broad applicability [18]. In this study, we investigated the impact of varying levels of AI confidence on trust, agreement, performance, and diagnosis duration within the healthcare domain to address RQ2.

Research shows that dynamic confidence displays can reduce automation bias. For example, [59] found that pilots who had access to real-time AI confidence updates were less likely to blindly follow system recommendations. Similarly, we found that low AI confidence significantly decreased trust and agreement but increased diagnosis duration. This cautious behavior aligns with [18, 17], who noted that lower confidence scores encourage users to rely more on their own judgment, leading to more thoughtful decision-making.

On the other hand, high confidence slightly increased trust, consistent with findings by [17, 18]. However, it also led to a small but significant decrease in performance, likely due to overreliance on AI recommendations. This overreliance, often referred to as automation bias, has been introduced by Parasuraman and Riley [11], which highlight how users tend to trust high-confidence AI outputs without critically assessing them.

These findings highlights the paramount importance of designing AI confidence displays to encourage appropriate reliance on AI systems. While high confidence can enhance trustworthiness, it must be presented in a manner that mitigates automation bias and ensures balanced decision-making, particularly in critical domains such as healthcare.

To address Research RQ3, we analyzed participant demographics and their perceptions of AI, revealing significant relationships between age, gender, job role, and user interaction with AI systems. Research indicates that individuals of different ages may adopt varied strategies when assessing the trustworthiness of automated systems [33]. In contrast to findings by Jermutus et al. [34], our results show that older participants rated the role and usefulness of AI more positively, whereas younger participants reported higher levels of mental demand and stress during interactions with AI.

Gender differences also emerged as a critical factor. Female participants perceived tasks involving AI as more complex and demanding and rated the usefulness of AI lower than their male counterparts, consistent with Tung’s [78] findings. These variations may be influenced by differences in how males and females respond to an automated system’s communication style and interface design [33, 78, 79].

Participants’ professional and technological backgrounds played a significant role in shaping their perceptions of AI. Research shows that users with higher digital literacy and greater exposure to AI systems are generally more comfortable and confident in adopting such technologies [80]. Radiologists, for instance, viewed AI as more integral to their work compared to other professional groups, likely due to their frequent exposure to AI systems specifically designed for radiology tasks. In contrast, oncologists found tasks involving AI to be more complex and demanding. This may reflect the relative lack of AI tools tailored to their specific workflows or the need for additional training to bridge gaps in familiarity and confidence. Social influences, including peer and family attitudes toward AI, also play a role in shaping users’ willingness to engage with these systems [25].

While this study offers valuable insights, there are some important limitations to consider. Because the experiment was conducted in a controlled setting, it doesn’t fully reflect the complexities of real-world clinical environments. Factors like emotional involvement, high-pressure decision-making, and how the system fits into existing workflows weren’t part of this study. We also relied on self-reported measures like trust and cognitive load, which may not always match what participants actually think or do. Additionally, the participant group wasn’t large or diverse enough to capture the wide range of perspectives found in real-world clinical settings.

Future research should explore how these findings apply to different environments and longer-term use. By addressing these gaps, we can better understand how to design AI systems that are effective, trustworthy, and equitable in practice.