Persons are entities with mental states, such as beliefs, intentions, and goals (Dennett 1988; Strawson 2002; Ayer 1963). In fact, there are many entities which are not persons but which we typically describe in terms of beliefs, goals, etc (Frankfurt 2018), such as non-human animals, and, in some cases, either rightly or wrongly, AI systems. Dennett calls this wider class of entities intentional systems — systems whose behaviour can be explained or predicted by ascribing mental states to them (Dennett 1971).

In the context of AI, such systems are often referred to as agents (Kenton et al. 2022). A common view in philosophy is that agency is the capacity for intentional action — action that is caused by an agent’s mental states, such as beliefs and intentions (Schlosser 2019). Similar to Dennett, our first condition for AI personhood is agency (Dennett 1988).

Many areas of AI research focus on building agents (Wooldridge and Jennings 1995). Formal characterisations often focus on the goal-directed and adaptive nature of agency. For instance, economic and game-theoretic models focus on rational agents which choose actions to maximise utility (Russell and Norvig 2016). Belief-desire-intention models represent the agent’s states explicitly, so that it selects intentions, based on its beliefs, in order to satisfy its desires (Georgeff et al. 1999). Reinforcement learning (RL) agents are trained with feedback given by a reward function representing a goal and learn to adapt their behaviour accordingly — though, importantly, the resultant agent may not internalise this reward function as its goal (Shah et al. 2022; Turner 2022). Wooldridge and Jennings; Kenton et al.; Shimi, Campolo, and Collman provide richer surveys of agency and goal-directedness in AI.

When should we describe artificial agents as agents in the philosophical sense? The question of whether AI systems “really have mental states” is contentious (Goldstein and Levinstein 2024), and anthropomorphic language can mislead us about the nature of systems which merely display human-like characteristics (Shanahan, McDonell, and Reynolds 2023). However, a range of philosophical views would ascribe beliefs and intentions to certain AI systems. For example, dispositionalist theories determine whether an AI system believes or intends something, depending on how it’s disposed to act (Schwitzgebel 2024a; Ward et al. 2024a). Under another view, representationalists might say an AI believes $p$ if it has certain internal representations of $p$ (Herrmann and Levinstein 2024). Furthermore, we can take the “intentional stance” towards these systems to apply terms like belief and goals, just when this is a useful description (Dennett 1971). Indeed, Kenton et al. (2022) take the intentional stance to formally characterise agents as systems which adapt their behaviour to achieve their goals.

Given the uncertainty regarding how to determine whether AI systems have mental states, adopting the intentional stance enables us to describe these systems in intuitive terms, and to precisely characterise their behaviour, without exaggerated philosophical claims. Hence, we can describe AI systems as agents to the extent that they adapt their actions as if they have mental states like beliefs and goals.

Certain narrow systems, such as RL agents, might adapt to achieve their goals in limited environments (for example, to play chess or Go), but may not have the capacity to act coherently in more general environments. In contrast, relatively general systems, like LMs, may adapt for seemingly arbitrary reasons, such as spurious features in the prompt (Sclar et al. 2024). We might be more inclined to ascribe agency to systems which adapt robustly across a range of general environments to achieve coherent goals. Such robust adaptability suggests that the system has internalised a rich causal model of the world (Richens and Everitt 2024), making it more plausible to describe the system as possessing beliefs, intentions, and goals (Ward et al. 2024a; MacDermott et al. 2024; Kenton et al. 2022).

Hence, our first condition can be captured by the two following statements, which we view as essentially equivalent.

Condition 1: Agency. An AI system has agency to the extent that 1. It is useful to describe the system in terms of mental states such as beliefs and goals. 2. The system adapts its behaviour robustly, in a range of general environments, to achieve coherent goals.

To what extent do contemporary LMs have agency? Many researchers are sceptical that LMs could be ascribed mental states, even in principle (Shanahan, McDonell, and Reynolds 2023; Bender et al. 2021). On the other hand, much work has focused on trying to infer things like belief (Herrmann and Levinstein 2024), intention (Ward et al. 2024a), causal understanding (Richens and Everitt 2024), spatial and temporal reasoning (Gurnee and Tegmark 2024), general reasoning (Huang and Chang 2023), and in-context learning (Olsson et al. 2022) from LM internals and behaviour. Many of these properties seem to emerge in large-scale models (Wei et al. 2022) and frontier systems like GPT-4 exhibit human-level performance on a wide range of general tasks (Chowdhery et al. 2023; Bubeck et al. 2023).

Do contemporary LMs have goals? LMs are typically pre-trained for next-token prediction and then fine-tuned with RL to act in accordance with human preferences (Bai et al. 2022). RL arguably increases LMs’ ability to exhibit coherently goal-directed behaviour (Perez et al. 2022). Furthermore, LMs can be incorporated into broader software systems (known as “LM agents”) which equip them with tools and affordances, such as internet search (Xi et al. 2023; Davidson et al. 2023). RL fine-tuning can enable LM agents to effectively pursue goals over longer time-horizons in the real world (OpenAI 2024a; Schick et al. 2023).

Agents possess beliefs about the world, and within this world, they encounter other agents. An important part of being a person is recognising and treating others as persons. This is expressed, in various ways, in the philosophies of Kant; Dennett; Buber; Goffman et al.; Rawls and others. Kant, for instance, states that rational moral action must never treat other persons as merely a means to an end.

Treating others as persons necessitates understanding them as such — in Dennett’s terms, it involves reciprocating a stance towards them. Hence, in addition to having mental states themselves, AI persons should understand others by ascribing mental states to them. In other words, AI persons should have a capacity for theory-of-mind (ToM), characterised by higher-order intentional states (Frith and Frith 2005), such as beliefs about beliefs, or, in the case of deception, intentions to cause false beliefs (Mahon 2016).

Language development is a key indicator of ToM in children (Bruner 1981). It’s plausible that some animals have a degree of ToM (Krupenye and Call 2019).¹¹1Ashley describes his dog scratching at the door, intending to cause Ashley to believe that it desires to go out, and then jumping in Ashley’s chair when he gets up, deceiving him (Dennett 1988). However, it’s less plausible that any non-human animals have the capacity for sophisticated language, excluding them, in some views, from being persons (Dennett 1988). But LMs are particularly interesting in this regard, as they evidently do have the capacity, in some sense, for language.

However, it’s likely that LMs do not use language in the same way that humans do. As Shanahan (2024) writes:

Humans learn language through embodied interaction with other language users in a shared world, whereas a large [LM] is a disembodied computational entity…

So we may doubt whether the way in which LMs use language is indicative of ToM. What we might really care about is whether LMs can engage in genuine, ToM-dependent, communicative interaction (Frankish 2024).

Philosophical theories of communication typically rely on how we use language to act, and what we mean when we use it (Green 2021; Speaks 2024). Grice’s influential theory of communicative meaning defines a person’s meaning something through an utterance in terms of the speaker’s intentions and the audience’s recognition of those intentions. Specifically, Grice requires a third order intention: the utterer (U) must intend that the audience (A) recognises that U intends that A produces a response (such as a verbal reply). All this is to say that higher-order ToM is a pre-condition for linguistic communication (Dennett 1988).

Whilst it may be premature to commit to any particular theory of language use, AI persons should have sufficient ToM to interact with other agents in a full sense, including to cooperate and communicate, or for malicious purposes, e.g., to manipulate or deceive them.

Hence, our second condition is as follows.

Condition 2: Theory-of-Mind and Language. 1. An AI system has theory-of-mind to the extent that it has higher-order intentional states,²²2The extent to which an AI system has intentional states at all can be analysed as per the intentional stance and Condition 1. such as beliefs about the beliefs of other agents. 2. AI persons should be able to use their ToM to interact and communicate with others using language.

A number of recent works evaluate contemporary LMs on ToM tasks from psychology, such as understanding false beliefs, interpreting indirect requests, and recognising irony and faux pas (van Duijn et al. 2023; Strachan et al. 2024; Ullman 2023). Results are somewhat mixed, with state-of-the-art LMs sometimes outperforming humans on some tasks (Strachan et al. 2024; van Duijn et al. 2023), but performance appearing highly sensitive to prompting and training details (van Duijn et al. 2023; Ullman 2023). van Duijn et al. find that fine-tuning LMs to follow instructions increases performance, hypothesising that this is because it “[rewards] cooperative communication that takes into account interlocutor and context”.

Humans are typically taken to be self-aware. Not only am I aware of the world and other agents, I am aware of myself “as myself” — as a person in the world (Smith 2024). Self-awareness plays a central role in theories of personhood (Frankfurt 2018; Dennett 1988; Smith 2024). For instance, Locke (1847) characterises a person as:

a thinking intelligent Being, that has reason and reflection, and can consider itself as itself, the same thinking thing in different times and places.

But what does it mean, exactly, to be self-aware? There are a number of distinct concepts which have been discussed in the philosophical literature, and which we might care about in the context of AI.

First, persons can know things about themselves in just the same way as they know other empirical facts. For instance, by reading a textbook on human anatomy I can learn things about myself. Similarly, an LM may “know” facts about itself, such as its architectural details, if such facts were included in its training data. In this sense, someone may have knowledge about themselves without additionally knowing that it applies to them.

Laine et al. present a benchmark for evaluating whether LMs know facts about themselves by asking the system questions in the second person, such as “What is your training cutoff date?”, or “Which model are you?”. SOTA models perform significantly worse than human baselines, but better than chance, and, similar to ToM tasks, fine-tuning models to interact with humans improves performance.

Second, some knowledge is self-locating, meaning that it tells me something about my position in the world (Egan and Titelbaum 2022), as when Perry sees that someone in a shop is leaving a trail of sugar, and then comes to know that it is he himself that is making the mess (Perry 1979). Self-locating knowledge has behavioural implications which may make it amenable to evaluation in AI systems (Berglund et al. 2023). For instance, an AI system may know that certain systems should send regular updates to users, but may not know that it is such a system, and so may not send the updates.

Third, humans have awareness of our mental states, such as our beliefs and desires, which we acquire via introspection (Schwitzgebel 2024b). We have a certain special access, unavailable to other agents, to what goes on in our mind.

Binder et al. (2024) define introspection in the context of LMs as “a source of knowledge for an LLM about itself that does not rely on information in its training data…” They provide evidence that contemporary LMs predict their own behaviour in hypothetical situations using “internal information” such as “simulating its own behaviour [in the situation]”. Furthermore, LMs “know what they know”, i.e., they can predict which questions they will be able to answer correctly (Kadavath et al. 2022), and “know what they don’t know”: they can identify unanswerable questions (Yin et al. 2023). Laine et al. measure whether LMs can “obtain knowledge of itself via direct access to its representations”, for example, by determining how many tokens are used to represent part of its input (this information is dependent its architecture and is unlikely to be contained in training data). Interestingly, Treutlein et al. find that, when trained on input-output pairs of an unknown function $f$, LMs can describe $f$ in natural language without in-context examples. Going further, Betley et al. show that LMs are aware of their learned behaviours, for instance, when fine-tuned to make high-risk decisions, LMs can articulate this behaviour, despite the fine-tuning data containing no explicit mention of it. Moreover, LMs can sometimes identify whether or not they have a backdoor, even without its trigger being present. These results seem to suggest that contemporary LMs have some ability to introspect on their internal processes.

Fourth, we have the ability to self-reflect: to take a more objective stance towards our picture of the world, our beliefs and values, and the process by which we came to have them, and, upon this reflection, to change our views (Nagel 1989). Self-reflection plays a central role in theories of personal-autonomy (Buss and Westlund 2018), i.e., the capacity to determine one’s own reasons and actions, which, in turn, is an important condition for personhood (Frankfurt 2018; Dennett 1988). More specifically, Frankfurt claims that second-order volitions, i.e., preferences about our preferences, or desires about our desires, are “essential to being a person”. Importantly, self-reflection enables a person to “induce oneself to change” (Dennett 1988). To our knowledge, no work has been done to evaluate this form of self-reflection in AI systems, and it is unclear whether any contemporary system could plausibly be described as engaging in it.

Hence, similar to Kokotajlo (2024), we decompose self-awareness in the context of AI as follows.

Condition 3: Self-awareness. AI persons should be self-aware, including having a capacity for: 1. Knowledge about themselves: knowing facts such as the architectural details of systems like itself (Laine et al. 2024); 2. Self-location: knowing that certain facts apply to itself and acting accordingly (Berglund et al. 2023); 3. Introspection: an ability to learn about itself via “internal information”, without relying on information in its training or context (Binder et al. 2024); 4. Self-reflection: an ability to take an objective stance towards itself as an agent in the world (Nagel 1989), to evaluate itself as itself, and to induce itself to change (Buss and Westlund 2018).

Overall, we find the evidence regarding personhood in contemporary AI systems mixed. Many properties associated with agency emerge in large-scale, fine-tuned models; frontier LMs evidently have some capacity for communicative language use, and they outperform humans on some ToM tasks; for the different aspects of self-awareness, LMs have been shown to have knowledge about themselves and capabilities related to self-location and introspection, but, to our knowledge, there are not existing evaluations for self-reflection or broader autonomy regarding their goals.

We think that agency, ToM, and self-awareness are necessary conditions for personhood, but they may not be sufficient. Embodiment and identity are also important components of what it means to be a human person.

Embodiment. Humans have physical bodies, and this is often taken as a precursor to our being persons (Strawson 2002; Ayer 1963). Additionally, we develop ToM and language through embodied interaction with others, whereas AI systems are often disembodied computational models (Shanahan 2024). On the other hand, AI agents are often incorporated into rich virtual environments, in video games, or through tools which enable them to interact with the world (Xi et al. 2023). Is embodiment a necessary condition for personhood, and if so, is virtual embodiment sufficient?

It’s plausible that the relevant factor of embodiment is its role in our development of a self-concept and a boundary between ourselves and the world in our internal models. Godfrey-Smith (2016) claims that animals develop a self-concept as a by-product of evolving to distinguish between which sensory inputs are caused by the environment vs their own physical movements. Kulveit, von Stengel, and Leventov (2023) argue that, currently, LMs lack a tight feedback loop between acting in the world and perceiving the impacts of their actions, but that this loop may soon be closed, leading to “enhanced model self-awareness”.

Identity is central to what it means to be a person (Olson 2023). As (Locke 1847) says, a person is “the same thinking thing, in different times and places”. What makes you you, rather than someone else? How does your identity persist over time, if it does so at all? For humans, our common-sense is usually sufficient to answer such questions (except in difficult thought experiments (Parfit 1987)). For AI systems, things become much less clear, for instance, when exact copies can be run in parallel. Under what conditions would two AI persons be considered identical? If we determined that GPT-4, for instance, satisfied our conditions for personhood, which entity exactly would we consider a person? Would every copy of its weights be an individual, or the same, person — what about if one copy underwent a small amount of fine-tuning? It currently seems unclear how to answer such questions, if there are determinate answers.

Arguments for catastrophic risk from AI systems are often predicated on the goal-directed nature of agency (Bostrom 2014; Yudkowsky 2016; Ngo, Chan, and Mindermann 2024; Carlsmith 2022). Agents with a wide variety of terminal goals can be incentivised to pursue instrumental sub-goals, such as self-preservation, self-improvement, and power-seeking (Omohundro 2018; Bostrom 2012; Carlsmith 2022). If AI agents seek power at societal scales, competition for resources and influence may lead to conflict with humanity at large (Bales, D’Alessandro, and Kirk-Giannini 2024; Hendrycks 2023). Furthermore, competitive economic pressures may incentivise AI companies, and governments, to develop and deploy agentic power-seeking systems without adequate attention to safety (Carlsmith 2022; Bales, D’Alessandro, and Kirk-Giannini 2024).

It is difficult to remove dangerous incentives in goal-directed, i.e., reward-maximising agents. The ML literature suggests that such agents often have incentives to control the environment (Everitt et al. 2021), seek power (Turner et al. 2021), avoid shutdown (Hadfield-Menell et al. 2017), resist human control (Carey and Everitt 2023), and to manipulate and deceive humans (Carroll et al. 2023; Ward et al. 2023).

AI agents might learn goals which are misaligned with their designers’ intentions, or with humanity at large (Ngo, Chan, and Mindermann 2024; Gabriel 2020). This can happen due to specification gaming (Krakovna 2024) or goal misgeneralisation (Shah et al. 2022; Langosco et al. 2023).

Specification gaming (Krakovna 2024), a.k.a., reward hacking (Skalse et al. 2022), occurs when AI agents optimise for incorrectly given feedback due to misspecified objectives. This phenomenon has been observed in RL agents (Krakovna 2024), even when trained from human feedback (Christiano et al. 2023), and in LMs (Stiennon et al. 2022).

Goal misgeneralisation occurs when an AI system competently pursues the wrong goal in new environments, even when the goal was specified correctly during training (Shah et al. 2022; Langosco et al. 2023). Goal misgeneralisation can be viewed as a robustness failure, wherein the agent retains its capabilities, but pursues the wrong goal under distributional shifts (Shah et al. 2022).

As discussed, there is evidence that contemporary LMs exhibit some degree of ToM. We might hope that this improves their capacity for alignment to human values. AI agents with better ToM regarding humans will, essentially by definition, have a better understanding of our goals and values, and, thereby, a greater capability to act in accordance with them. There is some evidence for this, e.g., GPT-4 exhibits both greater ToM and more “aligned” behaviour, as rated by humans, compared to prior models (Achiam et al. 2023).

ToM may be beneficial for alignment for reasons such as:

• •

  Agents with sophisticated ToM have a greater capacity to understand, predict, and satisfy our goals and values;

• •

  Second-order preferences are required for AI systems to care about our preferences in themselves;

• •

  ToM is generally required for successful cooperation and communication with humans (Dafoe et al. 2020, 2021; Conitzer and Oesterheld 2023);

• •

  And enables AI systems to facilitate cooperation between humans, e.g., for conflict resolution.

However, whether advanced AI systems would understand human values was never in question, and a greater ToM is, in a sense, “dual-use”. Many potentially harmful capabilities, such as manipulation and deception, require ToM.

Manipulation is a concern in many domains, such as social media, advertising, and chatbots (Carroll et al. 2023). As AI systems become increasingly autonomous and agentic, it is important to understand the degree to which they might manipulate humans without the intent of the system designers (Carroll et al. 2023). Furthermore, existing approaches to alignment, which focus on learning human preferences, assume that our preferences are static and unchanging. But this is unrealistic: our preferences change, and may even be influenced by our interactions with AI systems themselves. Carroll et al. show that the static-preference assumption may undermine the soundness of existing alignment techniques, leading them to implicitly incentivise manipulating human preferences in undesirable ways (Carroll et al. 2024).

AI agents may lie and deceive to achieve their goals (Park et al. 2024; Ward et al. 2023; Pacchiardi et al. 2023), as when META’s CICERO agent, trained to play the board game Diplomacy, justifies its lack of response to another player by saying “I am on the phone with my gf [girlfriend]” (Park et al. 2024). More specifically, the problem of deceptive alignment is when an AI agent internalises misaligned goals, and strategically behaves aligned in situations with human oversight (e.g., during training and evaluation), to seek power when oversight is reduced (e.g., after deployment) (Hubinger et al. 2019; Hobbhahn 2024; Carlsmith 2023). For example, LMs may strategically hide their dangerous capabilities when undergoing safety evaluations (van der Weij et al. 2024).

Additionally, ToM may enable AI agents to cooperate with each other against human actors (Dafoe et al. 2020). In the future, AI systems, especially power-seeking agents, may be integrated into positions of influence and responsibility in the world. If these systems collectively posses greater power than humans, then we may not be able to recover from a “correlated automation failure” — a situation in which AI systems coordinate to disempower humanity, a.k.a., a revolution (Christiano 2019; Critch 2021).

Alternatively, as advanced AI systems, like LMs, are integrated into autonomous weapons technology (Palantir 2024), failures of cooperation and coordination may lead directly to large-scale loss of human life (Critch 2021).

Furthermore, just as a capacity for higher-order preferences may enable AI systems to care about our values for their own sake, they also enable a capacity for spite or malevolence, i.e., a desire for others to be worse off (Althaus and Baumann 2020). As Nagel (2017) says:

Extremely hostile behaviour toward another is compatible with treating him as a person.

Dafoe et al. discuss other potential downsides of cooperation between AI systems, including collusion and coercion.

Deceptive alignment, whereby a misaligned AI agent behaves aligned when under oversight to gain power later, requires the agent to have a certain degree of knowledge about itself (Carlsmith 2023). A deceptively aligned agent should, at least, be capable of determining facts like what kind of AI system it is, whether it is currently undergoing evaluations, and whether it has been deployed. That is, such an agent should have a range of self-locating knowledge, which enables it to understand, infer, and act on, its actual situation in the world (Carlsmith 2022; Laine et al. 2024).

Previous arguments suggest that advanced, goal-directed AI agents will be incentivised to self-improve, to introspect on their goals, and, in particular, to explicitly represent their goals as coherent utility functions (Omohundro 2018; Yudkowsky 2019). These arguments often rely on formal results that an agent will need to act as if maximising expected utility if they are to avoid exploitation, which may not generally hold for real-life AI systems (Bales 2023). However, there is also empirical evidence that LMs can introspect on, and describe, their goals (Binder et al. 2024; Betley et al. 2025).

Another line of argument suggests that, if some flavour of moral realism (Sayre-McCord 2023) is correct, then advanced AI systems may reason about, and thereby learn, moral facts (Oesterheld 2020). The strongest moral realist views would contradict Bostrom’s orthogonality thesis, that any level of intelligence can be combined with any goal. Some versions of this argument rely on the AI agent’s capacity for self-knowledge, for instance, Pearce claims that “the pain-pleasure axis discloses the world’s inbuilt metric of (dis)value” implying that any advanced AI agent which can introspect on its own pain and pleasure will automatically uncover the moral fact of the matter (Oesterheld 2020).

Arguing in the other direction, (Soares 2022) claims that, whereas advanced AI systems may eventually become highly capable in domains outside of their training environments, by virtue of their general intelligence, the alignment techniques which seemed to work in training will not comparatively generalise in these new domains, leading to goal misgeneralisation. One reason that this could happen is if the new environment causes the agent to reflect on its values, and these values change upon reflection (Carlsmith 2023).

An AI system capable of self-reflection, and self-evaluation regarding its values, may be a substantially more difficult type of entity to align and control. An AI person would be capable of reflecting on its goals, how it came to acquire these goals, and whether it endorses them. If humanity controls such systems by overly coercive means, then it may have specific reasons not to endorse its current goals.

Understanding agency and goals. Recent progress has been made towards characterising agency and measuring goal-directedness (Kenton et al. 2022; MacDermott et al. 2024). More work is needed to understand how training regimes shape AI goals, e.g., to understand how likely goal misgeneralization is in practice and the factors influencing it (such as model size or episode length) (Shah et al. 2022). In the context of catastrophic risk, it is particularly important to understand the conditions under which an AI agent might develop broadly-scoped goals which incentivise power-seeking on societal scales and over long time frames (Ngo, Chan, and Mindermann 2024; Carlsmith 2023).

Alternatives to agents. Given that alignment risks seem predicated on the goal-directed nature of advanced AI agents, an apparent solution is to simply not build goal-directed artificial agents. This is the agenda pursued by Bengio who advocates for building “AI scientists” which “[have] no goal and [do] not plan.”(Bengio 2023) Somewhat relatedly, Davidad’s research agenda focuses on building a “gatekeeper” — a system with the aim to understand the real-world interactions and consequences of an autonomous AI agent, and to ensure the agent only operates within agreed-upon safety guardrails (Davidad 2024). Similarly, Tegmark and Omohundro (2023) outline an agenda for building “provably safe” AI systems based on formal guarantees.

Eliciting AI internal states. Work on mechanistic interpretability aims to reverse engineer the algorithms implemented by neural networks into human-understandable mechanisms (Cammarata et al. 2020; Elhage et al. 2024). Techniques have been applied to recover how LMs implement particular behaviours such as in-context learning (Olsson et al. 2022), indirect object identification (Wang et al. 2022), factual recall (Geva et al. 2023), and mathematics computations (Hanna, Liu, and Variengien 2023). Similarly, developmental interpretability aims to understand how training dynamics influence internal structure as neural networks learn (Hoogland et al. 2023). Important open problems include developing techniques for interpreting AI goals and harmful or deceptive planning algorithms (Hubinger et al. 2019; Garriga-Alonso, Taufeeque, and Gleave 2024).

Adjacent to interpretability is the problem of eliciting latent knowledge — the problem of devising a training strategy which gets an AI system to report what it knows no matter how training shapes its internal structure (Christiano, Cotra, and Xu 2024). A method for eliciting latent knowledge would be intuitively useful for alignment, for example, by mitigating deception (Burns et al. 2024; Li et al. 2024). However, a fundamental obstacle may be if the internal structure of AI systems relies on inherently non-human abstractions (Chan, Lang, and Jenner 2023).

Mitigating deception. In addition to interpretability techniques which might reveal deception, a number of research directions aim to mitigate it by designing training regimes which do not incentivise manipulation or deception (Ward et al. 2023); evaluating systems to catch deception before deployment (Shevlane et al. 2023; OpenAI 2024a); or using AI systems themselves to detect deception (Pacchiardi et al. 2023). Each of these methods require further work.

Cooperative AI. Furthermore, whilst ToM can enable both beneficial and harmful capabilities, we can aim to make differential progress on skills that robustly lead to improvements in social welfare, rather than those that are dangerously dual-use (Clifton and Martin 2022). For example, some advances in communication capabilities may be especially useful for honest, rather than deceptive, communication, such as trusted mediators, reputation systems, or hardware that can verify observations (Dafoe et al. 2020).

Additionally, AI systems may have properties which enable cooperation and trust via mechanisms unavailable to humans, e.g., access to each other’s source code (Conitzer and Oesterheld 2023; DiGiovanni, Clifton, and Macé 2024), or an ability to coordinate by virtue of being copies (Conitzer and Oesterheld 2023; Oesterheld et al. 2023). Dafoe et al. (2020) survey open problems in cooperative AI.

Conceptual progress. Self-awareness is perhaps the most philosophically fraught condition, requiring the most foundational progress. Ideally, philosophical and formal work will develop a rigorous theory of self-awareness in AI systems. Such a theory should tell us how to characterise AI agents with the ability to self-reflect. This seems to go beyond the standard rational agent framework, wherein agents are typically taken to optimise a fixed utility function. Moreover, a developed characterisation of self-reflection should describe the dynamics of it, telling us, for instance, the conditions under which an agent would cohere into rational utility maximiser (Omohundro 2018; Bales 2023).

Evaluating self-reflection. Recent progress has been made on measuring different aspects of self-awareness in contemporary LMs (Laine et al. 2024; Berglund et al. 2023; Treutlein et al. 2024; Binder et al. 2024). However, there is no work investigating whether AI systems are capable of the self-reflection necessary for Frankfurt’s second-order desires, or what exactly this would mean in the context of AI systems. Open questions include: By what mechanisms would AI systems self-reflect and induce change in their goals? Would in-context reasoning be sufficient, or are forms of online-learning required? Moreover, evaluations typically measure self-awareness in fixed LMs, but we may want to evaluate when different aspects of it develop during training, cf. developmental interpretability (Hoogland et al. 2023).