Recentering Validity Considerations through
Early-Stage Deliberations Around AI and Policy Design

Organizations are rapidly adopting AI-based decision-making tools to augment human expert decisions in high-stakes settings like child maltreatment, criminal justice, and healthcare (De-Arteaga et al., 2021; Holstein and Aleven, 2021; Yang et al., 2016). Research and development efforts around these tools have aimed to help overcome resource constraints and limitations (such as inconsistencies and cognitive biases in human decision-making (Chouldechova et al., 2018; Kahneman et al., 2021; Levy et al., 2021). However, the in-situ use of AI-based decision-making tools has been met with significant contention (De-Arteaga et al., 2020; Green and Chen, 2019; Holstein and Aleven, 2021; Levy et al., 2021; Holten Møller et al., 2020). A growing body of research and media has surfaced ways in which AI-based decision-making tools have failed to produce value in practice, despite showing promising evaluation results prior to deployment (Yang et al., 2019; Kawakami et al., 2022). To address these concerns, research and policymaking efforts have increasingly focused on improving the downstream properties of AI models such as fairness, interpretability, or predictive accuracy. These efforts often begin with the assumption that the AI tool actually “works” and that its design is basically sound, apart from such concerns (Coston et al., 2022; Raji et al., 2022).

However, field studies of actual AI-based decision-making tools used in organizations today are beginning to surface fundamental challenges around the validity of these tools (e.g., whether the model does what it purports to do). In complex, real-world decision-making contexts, models are typically trained to predict an imperfect proxy for human decision-makers’ actual decision-making goals. For example, in child welfare, prior research discussed how frontline workers are required to make day-to-day decisions with an AI tool that predicted outcomes misaligned with their actual decision-making objectives, professional training, and legal constraints (Kawakami et al., 2022). While child welfare workers consider the immediate safety risks and harms to a child to make decisions about screening investigation, the dominant design for an AI tool in this domain uses long-term predictions of child placement out of their home. In healthcare, clinicians may make decisions about resource allocation for high-risk patients by assessing each patient’s immediate medical needs, while an AI tool may predict longer-term healthcare costs (Kerr, 1975). In these decision-making contexts, underlying model validity and value alignment challenges have far-reaching downstream impacts on broader organizational culture and community welfare (Brown et al., 2019; Cheng et al., 2021). While expert decisions in these settings are guided by considerations around existing legal systems, developers’ current processes for designing models often fail to meaningfully involve legal experts, policymakers, or decision-makers with direct expertise. Instead, considerations around how an AI tool may interface with policy are often discussed retrospectively, after the tool is designed or deployed (Jackson et al., 2014; Yang et al., 2023).

This status quo design process, scattering policy and design considerations across time and space, presents several challenges to ensuring the design of sufficiently valid AI tools in real-world social decision contexts. In many cases, it is not possible to fully address questions about the validity of an AI tool without also considering the design of both organizational and public policies that shape how the system will be used. For example, evaluating whether a design proposal for an AI-based risk assessment tool captures an appropriate notion of “risk” requires understanding legal definitions of “risk” and relevant policies governing how frontline decision-makers currently make decisions. Without considering interactions between technology design, law, and policy in the process of designing an AI tool’s objective function, the resulting AI tool may lack validity. In this case, early-stage conversations around policy and design could proactively prompt new evaluations that assess the face or construct validity of a proposed AI tool design, in the context of proposed or existing organizational policies. Beyond this specific example, there is a broader missed opportunity for communities of stakeholders (e.g., policymakers, frontline workers, leadership, developers) to proactively exchange and synthesize knowledge around validity to make design decisions that are both informed by, and inform, policy.

Properly addressing these challenges requires turning our attention to the earliest stages of model development and adoption: How can we refocus policy and research efforts around validity considerations, by promoting early-stage deliberations around how to design AI-based technologies and associated policies? Today, we lack effective, practical processes for proactively engaging policymakers, developers, and other stakeholders in fundamental questions around the design and governance of AI systems (e.g., whether a deployed AI system will actually do what it purports to do). In this position paper, we propose supporting early-stage, multi-stakeholder deliberations around the validity of proposed AI tools as a step towards designing better policies and technologies together.

A growing chorus of research has called for increased developer attention around AI validity concerns (sometimes discussed via related concepts such as “AI functionality”) as an essential first step towards ensuring the safety of AI deployments (Coston et al., 2022; Raji et al., 2022; Wang et al., 2022). However, much of this work still lives at the theoretical level, geared towards academic researchers. Grounding these considerations around validity into real-world design and policymaking settings requires a diverse pool of expertise – from an understanding of existing organizational processes, needs, and constraints around designing AI tools to tacit knowledge of opportunities and boundaries for informing policy. Relatedly, it is critical that such early-stage deliberations promote knowledge-sharing and synthesis across a wide range of relevant stakeholders, including policymakers, frontline workers, community members, developers, and organizational leaders. Through this piece, we invite researchers, designers, and practitioners to explore anticipated challenges and opportunities to operationalizing these multi-stakeholder early-stage deliberations for policy and design.

We invite the human-AI interaction, science and technology studies, machine learning, and other relevant communities to further knowledge around these topics:

Shifting power imbalances in and through collaborative design. Effective early-stage deliberations around policy and design requires engaging stakeholders (e.g., frontline decision-makers and community members) who are often left out of the model design process in the current status quo. In other words, implementing an effective deliberation process may also require shifting institutional power imbalances across stakeholders of the AI tool. At the same time, a deliberation process in itself may be structured to intentionally help shift power imbalances, for example, through the use of accessible and shared language or stakeholder-specific questions. How can we best shift power imbalances across different stakeholders varying in position and background, in the process of collaboratively designing policy and technology? What forms of imbalances cannot be nudged through deliberations, and how might other forms of support play a role?

Supporting evaluative and generative discourse. Deliberations about the validity of AI tools may need to be both evaluative and generative, to promote sufficient organizational buy-in and ensure resulting ideas produce more benefits than harms in practice. However, there may be tensions between different stakeholders and their (perceived) stances towards technical innovation versus evaluation. For example, there may be a (mis)conception that policies and laws constrain technical innovation, hindering effective conversation. How can we promote shared goals for collaborative policy and design discussions, while also valuing and leveraging differences in perspectives towards the role of AI innovation in a given context?

Connecting to local policymaking organizations. While early-stage deliberations may help identify and design new policies to complement technology design, they do not ensure that such policies are actually implemented post-deliberation. How can we better connect organizations with local policymaking groups, to help streamline outputs resulting from designing policy and technology together?

Overcoming incentive structures and pressures. In practice, there may be incentive structures, social pressures, or infrastructural barriers that hinder different stakeholders’ desires or abilities to engage in substantive discourse around designing more responsible technologies and policies. What role might other forces of higher power (e.g., regulation) play in ensuring that process-oriented solutions like supporting early-stage deliberations have sufficient teeth in practice?

Exploring other design opportunities for recentering validity. Supporting early-stage deliberations is one possible solution for recentering validity considerations in policy and design discourse. However, it may not be the best solution. What might other opportunities for refocusing validity considerations in design and policy look like?