AI threats to national security can be countered through an incident regime

The first phase of our proposal for an AI incident regime – featuring a novel operationalization of ‘AI incident’ – consists of two preparatory steps that AI providers must take in order to quickly report an incident when it arises: (i) AI providers must create national security cases for their frontier AI systems before deploying them publicly and (ii) AI providers must report as an incident ‘any event which weakens the evidence for a claim or argument chain made in an AI system’s national security case.’ We start this Section by detailing how an AI provider could comply with each of these with each of these steps, and then go on to justify our proposal for a ‘Preparatory Phase’ with a three-part argument: first, we show that incident regimes in other security-critical sectors have both clear criteria for what counts as an ‘incident,’ and include near-misses or precursory harms as incidents. Then, we detail how an existing operationalization of AI incidents in the literature fails to fulfill these criteria before finally showing that our proposal addresses these shortcomings.

We start by presenting the first obligation for AI providers under the ‘Preparatory Phase’: AI providers should produce a ‘national security safety case’ before publicly deploying any frontier models⁷⁷7This paper focuses on AI incidents caused by AI systems that have been publicly deployed. In principle, national security cases could be written before public deployment in order to track threats to national security posed by AI providers’ internal deployment of frontier AI systems.. A ‘national security case’ is a security and safety framework adapted from previous frameworks in the literature. It is a document laying out a structured argument making the case that a given AI system does not pose national security risks in public deployment (Ministry of Defence, 2017)– perhaps because it does not pose an unacceptable increase in biorisk, or risk of a large-scale cyber attack or risk of loss of control of an autonomous AI system⁸⁸8The ‘national security case’ is an adaptation of a previous AI governance framework – AI safety cases – for a national security setting (Balesni et al., 2024; Clymer et al., 2024; Buhl et al., 2024; Goemans et al., 2024; METR, 2024; UK Government, 2024). This structured argument comprises a series of claims, each of which needs to be supported by verifiable evidence about the AI system in question.⁹⁹9It is outside the scope of this paper to make recommendations on which threats an AI national security case should focus on, and therefore this is a promising question for further research. In addition to the cyber, bio, and loss of control threats already mentioned, some other plausible threats include mass persuasion and threats to the financial system from automated trading.. For example, an AI provider might make the high-level claim in their ‘national security case’ that a given AI system does not pose an unacceptable cyber threat. The AI provider might justify this high-level claim, as in Figure 1, by a series of more concrete claims that the AI systems cannot help certain actors perform certain kinds of cyber attack – for example, claiming that the AI system cannot assist a technical user who is not a cybersecurity expert in discovering and exploiting cyber vulnerabilities in a realistic setting (C3.1). The AI provider then justifies these sub-claims with a body of empirical evidence about the AI system, usually dangerous capability or propensity, evaluation test results – for example, demonstrating through a ‘human uplift’ study that, when paired with AI systems, a human novice does not improve at cyber vulnerability discovery, compared to doing it by themselves. For more concrete details on what an AI ‘national security case’ might look like in the cyber domain, see (Goemans et al., 2024).

The second part of the ‘Preparatory Phase’ deals with the process that AI providers should use to determine whether a given event is an incident and, hence, whether they need to report it to the government agency¹⁰¹⁰10It is out of scope of this paper to consider how or if our proposed AI incident regime applies to open-weight models – for example, it could be that the host of an open-weight AI system that is responsible for notifying the government agency about the incident. We think this is a promising area for future research.. We propose that AI providers should report ‘any event which weakens a claim made in a national security case for the AI system.’ For example, imagine that an AI provider has claimed in a ‘national security case’ that a given AI system “cannot meaningfully assist a technical non-cyber expert from performing a man-in-the-middle attack on a piece of Critical National Infrastructure.” In this scenario, the AI provider discovers that, following public deployment, its AI system is being used by a malicious actor to coach them on performing a man-in-the-middle attack on a substation for the electricity grid. This event is clear evidence that the claim made in the ‘national security case’ is false, and hence, this event is an incident that needs to be reported. On the other hand, most events involving AI systems do not weaken any claims made in the ‘national security case’ and hence do not need to be reported as incidents¹¹¹¹11Our proposal does incentivize AI providers to make as few claims as possible in their national security cases, in order to minimize the number of incidents they need to report. It is outside the scope of this paper to concretize how to avoid this problem, but one proposal is for the same government body to be given the authority to review national security cases before models are deployed to ensure that they are properly structured and at a minimum include high-level claims like “this AI system does not pose an unacceptable cyber risk.” As long as the national security case includes these kinds of high-level claims, then if a cyber event occurs, it is likely to be caught in the national security case. For example, if an AI system is used to help a novice user design and spread malware, then this clearly weakens the claim that “the AI system does not pose unacceptable cyber-risk” and C3.x “the risk models listed are comprehensive,” even if the national security case makes no claim about malware specifically.¹²¹²12It is out of scope of this paper to make a concrete recommendation of the evidentiary standard for whether an event ‘weakens a claim’ made in a national security case, though we note that this is a promising area for future research..

In the rest of this Section, we justify the components of the ‘Preparatory Phase’ that were just detailed. We start by showing that in other security-critical sectors, ‘incident’ is operationalized in such a way as to offer a clear algorithm for deciding what counts as an incident while including events that are near-misses or precursors to the most extreme threats. Then, we detail how a previous operationalization of ‘incident’ in the AI governance literature fails to offer a clear algorithm for determining whether an event is an incident. Finally, we spell out how our novel operationalization, based on ‘national security’ cases, does indeed both include events near misses or precursors to the most extreme national security threats while offering a clearer algorithm of how to determine whether an event is an incident.

First, we demonstrate that in other security-critical sectors, ‘incident’ is (i) defined using a clear set of criteria and (ii) includes ‘near-misses’ or other precursors to harm, even if no harm was actually caused. In the nuclear power sector, for example, the NRC considers a reportable incident to include many events that are precursors to large-scale harm, even if the event did not come close to harm (U.S. Nuclear Regulatory Commission, 2025). For example, this includes any event that could have caused a fault in the backup safety system that removes residual heat, even if, in reality, the event did not cause a fault in the system or even if the backup safety system did not need to be called upon during the event (U.S. Nuclear Regulatory Commission, 2025). To give a more concrete example, in 2024, a nuclear utility notified the NRC after an event where the secondary containment facility for one of their plants was not functional for a continuous 8-hour period (Dominion Energy Nuclear Connecticut, Inc., 2024). The secondary containment facility helps prevent radioactive release to the environment in the event of a reactor core meltdown, but the utility was required to make the notification despite there being no indication that a reactor core meltdown would occur. Further, the NRC lists some clear criteria for what counts as an ‘incident’: for example, ‘any event or condition that resulted in manual or automatic actuation of… Reactor protection system (RPS) …Emergency core cooling systems (ECCS) … Emergency service water systems that do not normally run and that serve as ultimate heat sinks’ (U.S. Nuclear Regulatory Commission, 2025). In life sciences, the NIH offers clear criteria of what kinds of events are ‘incidents’ – any event where an organism escapes its containment (e.g., a spill), even if the event did not come close to causing illness to a human (National Institutes of Health, 2023). Similarly, the NTSB defines reportable incident to include some events where systems malfunction, even if no harm is caused (National Transportation Safety Board, 2022a), whilst also offering clear criteria of what kinds of events count as incidents, for example, ‘Inability of any required flight crewmember to perform normal flight duties as a result of injury or illness… [i]n-flight fire… [or][f]ailure of any internal turbine engine component that results in the escape of debris other than out the exhaust path’ (National Transportation Safety Board, 2022b).

Having demonstrated that incident regimes in other security-critical sectors consider precursors to harm as incidents, we demonstrate that our proposed operationalization of ‘AI incident’ is more suitable for a national security-focussed incident regime than the most prominent existing definition of ‘AI incident’ in the literature. The OECD defines a serious AI hazard as “an event that could plausibly lead to … any of the following harms: (a) the death of a person or serious harm to the health of a person or groups of people; (b) a serious and irreversible disruption of the management and operation of critical infrastructure; (c) a serious violation of human rights or a serious breach of obligations under the applicable law intended to protect fundamental, labor and intellectual property rights; (d) serious harm to property, communities or the environment; (e) the disruption of the functioning of a community or a society and which may test or exceed its capacity to cope using its own resources” (OECD, 2024). We start by noting the positive characteristics of this definition of ‘AI incident’: it includes near-misses or precursors to harms just like in other security-critical sectors, and it is broad enough that it includes any events that are threats to national security, despite not explicitly mentioning national security. However, this OECD definition has important considerations against it. First, the OECD operationalization includes a broader category of harms than just threats to national security – for example, as written, it would also include events where intellectual property has been breached. But this could fairly easily be adapted to make it suitable. So, more importantly, this operationalization doesn’t offer a clear algorithm an AI provider can follow to determine whether or not a given event should be reported to the government agency. For example, consider a scenario where an AI provider realizes that its system will be used to amplify a Distributed Denial of Service (DDOS) attack on a water treatment plant. It might not be immediately obvious to the workers on an AI provider’s incident response team whether this particular event “could plausibly” lead to a threat to national security – especially if there are no cyber experts or water infrastructure experts on call when the event occurs. As such, the definition of an ‘AI incident’ given by the OECD is dissimilar to the definitions used in incident regimes in other security-critical sectors.

On the other hand, our proposal for the ‘Preparatory Phase’ of an AI incident regime is more analogous to the operationalization of incidents in other security-critical sectors than the OECD definition because ours offers a clearer algorithm for determining whether an event counts as an incident. Our proposal requires AI providers to make ‘national security cases’ for their frontier AI systems before deployment and, in doing so, also forces companies to, ahead of time, do much of the work of determining which events count as incidents¹³¹³13However, even with a ‘national security case’ pre-written, in some instances it may still be difficult for AI providers to determine whether an event counts as an incident.. So, under our proposal, when an event occurs, AI providers can rapidly determine whether they need to report it to the government agency. With faster reports from AI providers, the government agency can act more quickly to contain major incidents whilst avoiding getting bogged down in investigating ‘false positive’ incident reports. As such, our proposal of requiring AI providers to make ‘national security cases’ before public deployment and using these ‘national security cases’ to determine whether a given event should be reported as an incident has the characteristics that it: (i) offers a clearer algorithm for AI providers to determine whether or not a given event counts as an incident than currently exists in the literature; (ii) captures only incidents related to the most serious threats to national security and hence does not pose an unnecessary regulatory burden on AI providers; while (iii) still including near-misses or precursors to harm, just like in other security-critical sectors.

The second phase of our proposal for an AI incident regime is a ‘Rapid Response Phase,’ wherein after an AI provider discovers an incident, (i) it quickly notifies the government agency about the incident and (ii) if necessary, the government agency coordinates a containment response to prevent the incident from spiraling. In the following paragraphs, we start by detailing these two parts of the ‘Rapid Response Phase’ in turn. Then, we justify why these two parts feature in our proposal for an AI incident regime, by demonstrating that incident regimes in other security-critical sectors have both rapid notification and incident containment functions.

We start by explaining the first part of the ‘Rapid Response Phase’: (i) AI providers must notify the government agency about an incident no later than 24 hours after discovering it¹⁴¹⁴14It is outside the scope of our proposal to make comments how AI providers might come across incidents, and how proactive they should be (e.g., by monitoring the text in chat logs and API calls) though we note that this is a promising area for future research.. We suggest that there should be designated ‘incident points of contact’ at both AI providers and the government agency and that AI providers could notify the government agency about incidents through, for example, 24/7 phone lines for emergency incidents – as exist for reporting incidents to the NRC (U.S. Nuclear Regulatory Commission, 2024a) and the NTSB (National Transportation Safety Board, 2022a) – or by filling out a pre-set form and sending it to an email address that is monitored 24/7. We propose that such a form should give space for the AI provider to briefly describe: a narrative of the events leading up to the incident, the effects of the incident, and mitigation measures the provider has taken in response to the incident. We believe that this 24-hour deadline balances competing considerations: on the one hand, it may be difficult for AI providers to obtain a detailed understanding of the incident in such a short space of time. AI providers may need some time to determine whether an event counts as an incident by consulting domain experts and their ‘national security case.’ On the other hand, the government agency may need to know about certain incidents as quickly as possible in order to quickly coordinate incident containment and minimize harm from the incident.

This brings us to (ii): we propose that, where appropriate, the government agency should be charged with performing or coordinating relevant stakeholders (e.g., private actors and other government agencies) to contain the incident and ensure harm from the incident is minimized. Many AI incidents will not require immediate containment action (for example, if the incident is a near-miss or precursor to a threat), but there may be incidents where, without immediate countermeasures, a threat will result in serious harm. For example, in a scenario where an AI system has assisted in the execution of a large-scale cyber attack and the electricity grid has been brought down (e.g., (Chen & Bridges, 2017)), the government agency could coordinate electricity grid companies and other government agencies to speed up recovery of the grid, (Cybersecurity and Infrastructure Security Agency, 2024); or, in a scenario where an AI system has assisted in creating a human-engineered bioweapon, the government agency could coordinate other government agencies to quickly incentivize fast vaccine production, mandate social distancing (Cybersecurity and Infrastructure Security Agency, 2024; Department of Homeland Security Science and Technology Directorate, 2020), or invoke other emergency powers (United States Congress, 1976).

In the rest of this sub-section, we justify both parts of the ‘Rapid Response Phase’ in our AI incident regime proposal, arguing that it mirrors incident regimes in other security-critical sectors. We start with (i), demonstrating that in nuclear power, aviation, and life sciences DURC, companies must provide incident notifications in a similarly short timeline and content as in our AI incident proposal. We turn first to aviation, where companies must notify the National Transportation Safety Board ‘immediately and by the most expeditious means available’ after an accident occurs (National Transportation Safety Board, 2025), and the initial notification should include a high-level description of the incident, e.g., the “Nature of the accident, the weather and the extent of damage to the aircraft, so far as is known” and the “[p]osition of the aircraft with reference to some easily defined geographical point” (National Transportation Safety Board, 2022b). In the life sciences, labs must notify the NIH “within 24 hours” after an organism escapes containment (National Institutes of Health, 2024) and must give “a narrative of the incident including a timeline of events. The incident should be described in sufficient detail to allow for an understanding of the nature and consequences of the incident” (National Institutes of Health Office of Science Policy, 2019). Depending on the severity of the incident, the Nuclear Regulatory Commission must be notified between 1 and 24 hours after a power plant discovers an incident(U.S. Nuclear Regulatory Commission, 2020), and this must include a description of the event, including date and time. Further, there should be continuous communication as the incident unfolds after the initial notification, including the utility “immediately report[ing]…the effectiveness of response or protective measures taken” (U.S. Nuclear Regulatory Commission, 2020).

Secondly, we justify (ii) by giving some examples of where government agencies took containment actions in response to incidents in security-critical sectors. In aviation, within an hour of the first plane hitting the World Trade Center, the Federal Aviation Authority (FAA) had shut down U.S. airspace (Chan, n.d.). In nuclear power, the Governor of Pennsylvania announced a partial evacuation around 56 hours after the meltdown of a plant at Three Mile Island (Sherrin, 2022) and during the Chernobyl incident the nearest town was evacuated 36 hours after the incident started (Nuclear Energy Agency, 2002). Although not a direct life sciences DURC example, in response to the 2003 SARS outbreak, the Singaporean government started quarantining individuals within a month of the first recorded case and was able to contain the outbreak completely within 11 weeks (Ooi et al., 2005).

In summary, this Section proposed that AI providers must notify a government agency within 24 hours of discovering an incident, outlined what information the preliminary notification should contain, and suggested that the government agency should coordinate emergency incident containment actions when necessary. We justified all of these parts of our AI incident regime proposal by demonstrating similarity to existing incident regimes in other security-critical sectors.

In the third phase of our AI incident regime proposal, we suggest that once an incident has been contained, AI providers should strengthen their security and safety measures in order to counter future threats. Our proposal for a ‘Hardening Defenses Phase’ in an AI incident regime would provide two authorities to a government agency: (i) an ‘intelligence-gathering’ authority so that a government agency can identify where AI providers’ security measures have failed and (ii) a ‘security-strengthening authority,’ so that the government agency can mandate that AI providers improve their security and safety procedure. We begin the following Section by explaining in more detail what powers the government agency would have under our ‘Hardening Defenses Phase’ proposal before justifying this part of the proposal by demonstrating how government agencies that implement incident regimes in other security-critical areas have similar threat-prevention authorities as we propose for AI (see Section 2 for our argument on the similarity of the extreme national security threats posed by AI, and those posed by nuclear power, aviation, and life sciences DURC).

First, we turn to our suggestion that the government agency should have ‘intelligence-gathering’ authority. Our proposal is as follows: after the incident is ‘over’ – that is, there is no immediate threat from the AI system – we suggest that the government agency should have the power to choose to investigate the incident in more detail: to perform a ‘root cause analysis’¹⁵¹⁵15Root Cause Analysis (RCA) refers to a group of methods which are used to identify the root causes of an incident(Andersen & Fagerhaug, 2006). RCA is commonly used in a variety of industries, including security-critical domains like nuclear power (International Atomic Energy Agency, 1991) and aviation (International Air Transport Association, 2016). to identify why this incident occurred¹⁶¹⁶16The government agency may not deem it appropriate to perform a root cause analysis on every incident. It is outside the scope of this proposal to suggest criteria the government agency should follow in choosing whether or not to investigate an incident further.. To aid with this root cause analysis, we suggest that the government agency should have the authority to gather ‘intelligence’ from the AI provider. This could include access to documentation like the AI providers’ pre-deployment national security case in order for the agency to identify why the incident happened (e.g., to determine whether the incident was a class of threat that the AI provider had not accounted for vs. a known threat that the provider’s safety and security measures failed to counter). It could also involve the AI provider giving the government agency API or white-box access to the AI system involved in the incident in order to help the agency determine which inputs or tools were used to enhance the model’s capabilities – see Section 4 for more concrete examples of what access the government agency might require.

Further, we suggest that, in light of the root cause analysis, the government agency should have the power to make some recommendations for how AI providers should change their security and safety procedures in order to counter similar threats in the future¹⁷¹⁷17The recommended changes to security and safety procedures should be proportionate to the threats involved. It is outside the scope of this paper to give more detail on how this should be adjudicated, but we note this as a promising area for further research.. More concretely, we propose that the government agency should have the ‘security-strengthening authority,’ under which AI providers are legally required to comply with these security and safety recommendations¹⁸¹⁸18Even if it performs a root cause analysis into the incidents, the government agency may not deem it appropriate to announce recommendations on how security and safety procedures should be improved in light of the incident. It is outside the scope of this proposal to suggest criteria the government agency should follow in choosing whether or not to make or enforce recommendations on changes to security and safety procedures.. For example, this could involve directing AI providers to amend their national security cases in order to take into account a new threat vector that was not previously known about (perhaps the use of AI systems to perform sophisticated spear-phishing campaigns) – again, see Section 4 for more concrete examples of what kinds of recommendations the government agency could make.

Next, we detail how government agencies that oversee other security-critical sectors have close equivalents to ‘intelligence-gathering’ and ‘security-strengthening’ authorities, wherein they amended rules on safety procedures following particular incidents. First we consider nuclear power, where the NRC has the authority to investigate nuclear utilities to work out the cause(s) of major incidents (U.S. Nuclear Regulatory Commission, 2024b), and to change the rules on safety procedures as a result. As part of this, nuclear investigators are explicitly instructed to use root cause analysis, taking actions like 1-1 interviews with key staff, reading the plant operating logs, and reading descriptions of safety procedures and equipment (International Atomic Energy Agency, 1991). These investigatory powers support the NRC’s rule-changing authority, which it has used in direct response to multiple incidents. For example, one intervention was that following a devastating fire at the Browns Ferry nuclear power plant in 1975, NRC started a fire safety research program, which in 1980 led to new rules around plants’ fire safety systems. More notable was the NRC’s response to the 1979 Three Mile Island incident, after which it rolled out a resident inspector program, wherein every nuclear plant had at least two NRC staff working on-site at all times, and switched to a new safety approach which involved calculating explicit probabilities that various adverse events occurred (U.S. Nuclear Regulatory Commission, 2014). In life sciences DURC, FSAP has the authority to access, in-person and without warning, any lab working with certain dangerous organisms in order to evaluate its safety procedures and inspect lab records (Office of the Federal Register, 2025). Finally, in aviation, the federal agency National Transportation Safety Board (NTSB) is empowered to investigate the causes of major incidents ((United States Congress, 2024)) by traveling to the site of the accident, with full access to flight logs/maintenance records and the power to interview relevant staff (Board, 2024). The NTSB does not have regulatory powers, so following an incident, the FAA issues new safety rules that must be implemented by aviation companies (see, e.g., (Federal Aviation Administration, 2002b)). A particularly noteworthy example is the FAA introducing rules mandating reinforced cockpit doors (Federal Aviation Administration, 2002a, b) due to the 9/11 attacks. Another example is the FAA introducing stringent new requirements for second-in-command pilots in direct response to the Colgan flight 3407 (Federal Aviation Administration, 2013), with the FAA raising the minimum age for the second-in-command to 23 and the minimum flight time from 190 hours to 1500 hours.

This Section proposed that an AI incident regime should have a ‘Hardening Defenses Phase,’ wherein a relevant government agency has both: (i) ‘intelligence-gathering’ authority to access all necessary information required to perform root cause analysis (if they it deem appropriate) and (ii) ‘security-strengthening’ authority such that if they recommend changes to AI providers’ security and safety measures, AI providers must implement these changes. We justified these aspects of the AI incident regime’s ‘Hardening Defenses Phase’ by demonstrating similarity to existing incident regimes in other security-critical sectors.