Assessing AI Impact Assessments: A Classroom Study

In our quantitative data analysis, we did not run statistical significance tests. Instead, we opted to report summary statistics only and compare them with students’ responses to closed-form questions. Due to our small sample size and the nature of the study, our numerical reports should be understood as suggestive evidence (and not statistically significant claims) to contextualize the qualitative findings and inform hypothesis making for future larger-scale studies, conducted with a more representative sample of the target population. To analyze the free response questions, we adopted a thematic analysis approach where we qualitatively coded participants’ responses in a shared coding session. We conducted a bottom-up affinity diagramming process [3] to identify higher-level groups. We then compared the frequency of each code across participants’ experimental conditions, i.e., across the template or scenario conditions, to understand if themes were shared across or unique to a particular template or scenario condition.

Overall, most students entered into the AIIA exercise with high levels of both excitement and concern about potential uses of foundational generative AI models. While seemingly contradictory, being simultaneously excited and concerned is a common sentiment voiced by many politicians, experts, and regulators who share great concern about potential harms of AI systems, absent measures for accountability and governance [23]. After completing their AIIA, the class’ average excited rating slightly decreased (-$0.16$ points) from $4.11$ to $3.94$ (where a rating of $5$ indicates “very excited”), and concerned rating increased (+$0.32$ points) from $4.00$ to $4.32$ (where a rating of $5$ indicates “very concerned”). A greater proportion of students changed their concernedness rating after completing the AIIA: only 24% of students changed their excitedness rating, while 50% of students changed their concern rating. Of the students that changed their rating, $7$ out of $9$ students became less excited, and $15$ out of $19$ students became more concerned. We observed differences in how students’ self-reported excitement and concern changed for different instruments, and different scenarios (complete results in Appendix C), where students who were given the general-purpose use scenario and US CIO or Microsoft AIIAs had relatively larger increases in their concern. We hypothesize that there was a slightly greater effect on participants’ concern because existing impact assessments are primarily focused on articulating potential negative rather than positive impacts of proposed systems.

To survey students’ perceptions of who should be held accountable to mitigate potential impacts, we asked them to rate “the relative level of responsibility you imagine for ML experts (compared to other stakeholders) to address [the issues they identified]”. The class’ average rating increased ($+0.26$) from a score of $4.24$ to $4.50$ after completing the AIIA. $20$ (52% of all) students changed their response in their post-questionnaire, the majority ($15$) of whom reported an increased relative level of responsibility for ML experts. The extent to which students’ ratings increased varied across AIIA instruments, where the Canadian AIIA resulted in the largest average increase (+$0.42$), which we speculate may be because the Canadian AIIA had the most detailed and thorough mitigation sections (summarized in Appendix A). We present extended results of how ratings varied across conditions in Appendix C.2.

While each AIIA consisted of a series of questions that users must respond to, each AIIA instrument varied in its format – design decisions such as whether the AIIA is administered as a static form or dynamic website, or the expected response type of each question (e.g., multiple choice or free response). We summarize key findings that are either shared across instruments or bespoke to individual instruments:

• •

  The AIIA has open-ended questions that grant freedom, yet require creativity. All three AIIA instruments included open-ended questions that elicited free text responses. One student pointed out that due to this open-endedness, the quality of the final completed AIIA “depends heavily on the reliability and creativity of the person/team filling it out” (P1709, Microsoft). Another student also noted that the instrument required them to make several subjective judgments because several terms were under-specified: “it required a lot of qualitative assessments like those for safety and reliability and intelligibility that were not well-defined and could vary a lot depending on who fills out the AIIA and when” (P1750, Microsoft).

• •

  The AIIA has close-ended questions that hide nuance. Participants also critiqued questions that restricted responses to a specific form, such as answers to Yes/No questions about the AI system, or checklists and multiple choice questions that asked users to sort the AI into relevant categories. One participant explicitly called attention to a specific section of their instrument (Microsoft Section 5.2, “Goal Applicability”): “the […] section classifies the systems into only two classes (i.e., yes or no). This may not measure the systems well.” (P1816).

  The Canadian AIIA differed from the others in that it used users’ responses to closed-ended questions to calculate a numeric “raw impact score” and “risk score”, which were displayed at the bottom of the AIIA webpage (shown in Figure 2). These scores were then used to sort the AI into one of four “impact levels”, which determine the mitigation steps that the system owners are required to take. However, many participants stated that they did not understand how the numeric scores were calculated (P1081), or found them difficult to interpret (P1384, P1153). One participant expressed a desire to better understand the method used to calculate the risk score: “we were confused about how points corresponded to our answers. It may be more helpful to be able to walk-through (after a first draft is completed) and see what alternative choices would result in a lower score” (P1081).

The three AIIA instruments had overlap, yet varied considerably in their content: the types of impacts and mitigation steps that users were prompted to consider. We provide a detailed comparison across instruments in Appendix A, and summarize participants’ critiques below:

• •

  The AIIA is missing types of impact. Several participants noted that they believed their AIIA was not comprehensive enough. Participants wrote that their instrument’s “categorized limitations and harms” are “non-extensive” (P1063, Canadian), or that their AIIA “doesn’t address broader issues beyond privacy and non-discrimination” (P1281, Microsoft). Another participant explicitly noted types of impact that they believed should have been included in their AIIA: “no question had us think critically about the accessibility of the system, or which subgroups/subpopulations could be disproportionately harmed if the model was to be deployed” (P1081, Canadian).⁴⁴4The Canadian AIIA has been updated to include additional questions regarding accessibility since the students participated in the study.

• •

  The AIIA includes questions that may not be applicable to all AI systems. Some participants commented that not all questions may be relevant to all AI systems. One participant that completed the Microsoft AIIA found that “some questions did not strictly apply to our product” (P1819). This calls into question whether it makes sense to assess all proposed AI systems with a common (versus more specialized) AIIA instrument, a tension that has been raised by past work [18, 20, 36].

While participants were instructed to complete the AIIA in a facilitated classroom exercise, several participants pointed out perceived or imagined obstacles that may prevent organizations from completing an AIIA in practice. We group these concerns as those related to the feasibility of the AIIA, i.e., what realistically must happen to effectively complete each questionnaire.

• •

  Users completing an AIIA desire supervision. Several students listed a lack of supervision (i.e., “no oversight”) as a limitation of the AIIA process (P1505, P1709). While all three instruments provided an email address that users could contact with questions or feedback on each instrument, only the Canadian AIIA listed named persons who could provide synchronous help and consultation.

• •

  Users completing an AIIA desire clear expectations and transparency about how their responses will be reviewed. Some students expressed uncertainty or confusion about how their completed AIIA would be evaluated or reviewed (P1613, P1709), or were “unsure if we answered the questions properly” (P1456). One student expressed concern specifically about how their open-ended responses would be evaluated: “a lot of the questions were short answer, which makes sense since there’s many different types of responses, but this sometimes made it unclear what should be answered” (P1613).

• •

  Users may lack information about the AI that is necessary for the AIIA at the time of its completion. Study participants were given a short one-paragraph description of a potential AI system. However, many students reported that this description did not provide enough information about the product to complete the AIIA (P1620, P1613, P1866, P1286, P1528). One participant commented that their AIIA “seems to be designed for more fleshed out products rather than just ideas” (P1286, Microsoft). Another participant’s team “didn’t have a lot of context about the development of the AI system itself, so when answering these questions, we made a lot of assumptions about use cases and the development process” (P1613, US CIO). When participants opted against making restrictive assumptions, they noticed that existing instruments did not allow them to consider multiple downstream possibilities: “some of the questions where we need to select only a single option to be hard to choose from since the product might apply to multiple options provided (for example, the role of humans in the product)” (P1819, Microsoft).

  We note that participants’ critiques may not only reflect potential limitations of the AIIA templates, but also of our study design (e.g., that students lacked knowledge or context about the imagined scenario that a practitioner would have). The scenario text (pasted in Appendix B.1) includes details of the proposed user interface but excludes implementation details such as the training dataset or model architecture, and was designed to simulate the information available in an AI services’ early ideation stages. Thus, we hypothesize that many of the above critiques may still be relevant for real AI systems that are at a similarly early phase of their design.

• •

  The AIIA requires coordinating a team of stakeholders with interdisciplinary expertise. Several participants reported that they lacked relevant knowledge or the appropriate domain expertise necessary to complete the AIIA. One participant that completed the US CIO AIIA was confused by “legality questions”, writing, “as technical members of the team, we were unfamiliar with the terminology used” (P1456). Another participant noted a “lack of explanation on some of the terms [and] regulations” (P1696, Canadian). This finding points to the practical need to assemble a team of stakeholders with appropriate interdisciplinary expertise, and identifying who is responsible for each piece. Existing AIIA interfaces presently do not include support for collaborating asynchronously across multiple devices.

Participants reflected on whether the AIIA was effective at helping them imagine, and develop a plan of action to mitigate potential harmful impacts of the proposed AI.

• •

  Potential harms & mitigation steps are vague or under-specified. Several participants reported that their AIIA used language that was “not clear” (P1145, P1505), “vague” (P1005, P1886), or “difficult to understand” (P1620). Unclear language throughout the AIIA posed challenges to helping users imagine and enumerate specific harms: “many of the questions were vague and did not go over the harms that could be caused” (P1886, Canadian). Some participants specifically called attention to their AIIA’s suggested mitigation steps, which they felt were under-specified: participants wrote that Microsoft’s AIIA “doesn’t provide specific guidelines on how to address risks” (P1281), “didn’t give us concrete directions of actions we need to take” (P1121), and “while good at identifying harms, [is] not very helpful in suggestions for mitigation” (P1286). One participant noted how this under-specification could potentially be exploited in favor of the organization completing the AIIA, thus making it less effective as an accountability mechanism: “many of the evaluations for mitigations were concerned with the existence of a procedure to handle something, but allowed this to be an in-house procedure and did not lay down any specifications as to what the procedure should consist of” (P1359, Canadian).

• •

  Existing AIIA instruments are less effective for general-purpose technologies. Across all three instruments, participants who were assigned to complete the AIIA for the general-purpose technology scenario reported a disconnect where parts of their AIIA did not seem applicable to general-purpose technologies. Participants that were shown the Canadian AIIA were especially more likely to note this disconnect (P1359, P1384, P1153, P1866, P1273), likely because the Canadian AIIA includes specific sections where the user must report the desired end use of the system (Section 7). Participants wrote that “the questionnaire seemed to be designed for someone both creating a model and applying it to a given application domain. It didn’t work as well for our case of creating a model that could be used in many domains” (P1359) and that “the AIIA does not account for the difference in scope of the project vs the potential applications” (P1153). Beyond critiques of existing instruments, one participant found imagining the potential impacts of a general-purpose system to be more challenging: “I think the major limitation is that the description of the product was vague so depending on which industry the product was being applied to there could be all sorts of different issues that AIIA could not really pick up” (P1866).