Optimal Behavior Prior:
Data-Efficient Human Models
for Improved Human-AI Collaboration

One of the ultimate goals of the field of AI is creating agents that are able to collaborate with us and help us achieve our goals. Such agents benefit from predictive models of human behavior – even if those models are simply used to simulate human behavior when training an RL policy [6]. Given the nuances and complexities of human behavior – our decisions being riddled with hundreds of identified cognitive biases [4, 20] – models of human behavior need to be grounded in real data.

At the same time, basing models solely on real human data has proven challenging. Human models tend to underperform relative to real humans [24] and fail to generalize to unseen or uncommon situations [33]. The generalization problem is exacerbated in collaboration, where using RL to compute an agent policy that is the best-response to the human model (i.e. the policy that is optimal for the AI agent given that the human will act according to the policy captured by the model) will create incentives for the RL system to exploit biases in the human model by exploring states during training that are from a different distribution than those induced by human-human collaboration.

Thus, the limited human data we have is very valuable, and should be used in the most efficient manner possible to learn nuances of human behavior. One way to do so might be to use low-capacity models of human behavior with high inductive bias, e.g. which leverage Theory of Mind [30, 8, 46, 21]. However, the quality of such models crucially depends on being able to add the correct inductive biases, and require extensive engineering effort. In contrast, we seek a solution that retains the expressivity of high-capacity models while addressing the data requirements from above.

Can we provide some structure for the model that, although useful in increasing the efficiency in our usage of human data, doesn’t reduce our model expressivity, i.e. its ability to improve to arbitrary levels of performance as the amount of data and the coverage increase? We hypothesize that one area with high potential for improvement is providing high-capacity human models with an informed prior about what human behavior should look like: generally, human models’ prior over human behavior (corresponding to their initialization) is random. But what makes for a good prior of human behavior? Our insight is that while human behavior deviates from optimality in nuanced ways which are hard-to-formalize, we should leverage the fact that people are at least trying to perform tasks optimally, and use optimal behavior as a prior.

This builds off of the intuition that for most human tasks, people’s behavior is closer to being optimal than to completely random behavior. Thus, this kind of approach will be all the more effective for settings in which human behavior is closer to optimal. But even if human behavior was not very optimal, we expect it to be much more sample-efficient to reduce a optimal model’s performance down to human-level (through human-data fine-tuning), rather than to learn to increase a random model’s performance up to being human-level. This kind of approach has two main advantages.

Firstly, it provides the model with a good starting representation for human behavior in the task: rather than wasting precious real human data on learning to represent basic aspects of the task (the prior takes care of that already) the learner can leverage the data for capturing the subtle nuances of human’s suboptimal behavior – following a similar idea behind pre-training and fine-tuning paradigms common in domains such as computer vision and NLP [31, 14].

Additionally, although our approach requires the extra component of access to optimal behavior, it is generally easier to come by than additional human data, which would otherwise be required for good human models: self-play methods [13] have been shown to work surprisingly well in obtaining near-optimal policies even in complex domains [36, 37, 44, 28]. To encode such an Optimal Behavior Prior (which we refer to as OBP), one can use the parameterization for near-optimal agents as a starting point for training one’s human model.

We test this idea in Overcooked-AI [6], a human-AI collaboration benchmark environment based on a video game by the same name [10]. In a simulated user study, we compare how different ways of modeling humans can affect a human models’ quality. We find that using OBP for our human models leads to qualitatively more realistic human behavior and much higher competence, enabling for the first time to obtain human models that generalize across different environments without being trained on specialized hard-coded state-featurizations. Additionally, we investigate whether such improved human model training regime can also improve downstream collaborative AIs trained using our human models. Again, we find that traditional behavior cloning models are not sufficiently realistic (in our data regime) to lead to good downstream collaborative AIs: collaborative agents obtained with simple behavior cloning perform even worse than self-play agents when paired with simulated humans. However, when using human models obtained with OBP for AI training, one can achieve better collaboration performance than under all other conditions.

Overall, our results show that OBP vastly increases human modeling data efficiency, suggesting that the technique could be widely applicable in any low-human-data human-modeling setting.

Human modeling. There has been a recent wave of interest in obtaining good human models for games [24, 18, 1]. One of the simplest approaches to imitation learning is given by behavior cloning, which learns a policy from expert demonstrations by directly learning a mapping from observations to actions using standard supervised learning methods  [3, 42]. This kind of approach has been historically the most successful at imitating human behavior [25, 7, 38]. However, one common problem with methods like behavior cloning is that such models tend to underperform relative to real humans [24], in part likely due to insufficient expert data. Recently, [18] was able to obtain more realistic human models by leveraging search to improve the human model performance, while using a behavior cloning policy regularization to keep the policy human-like. We see our approach as at least complementary to theirs: OBP models would likely constitute better human models for regularization with their method. However, OBP human models might even be competitive in isolation when compared to search-based human models, as the former will have better performance relative to traditional behavior cloning methods, potentially obviating the need for search.

Human-AI collaboration. In this paper, we particularly focus on the benefit that improving human models can have on improving Human-AI collaboration: building agents that can collaborate with humans [6, 27]. This is related to the ad-hoc coordination problem – coordination with unknown teammates [39, 2] – but differs from it in that we assume that the teammate will be human. While it has been claimed that one might be able to obviate the need for human data to collaborate with humans [40], this will not be sufficient in settings in which capturing nuances of human suboptimal behavior is essential for the tasks at hand. The knowledge that the partner will be a human can either be encoded directly through handcrafted inductive biases [16, 46, 15, 40, 29], or by leveraging human data to varying degrees. Some approaches use both human data and inductive biases: first identifying which Nash equilibrium humans tend to play, and then biasing agents trained in self-play to learn the same equilibrium [22, 43]. However, these approaches require making strong assumptions about what test-time human behavior will look like, which may not hold in practice. This has led to various approaches to achieving human-AI collaboration which rely on learning models of human behavior and computing best-responses to them [6, 17]. Our approach is closest to [6]: we first train a human model to capture human behavior, and then train a best response to it to obtain a collaborative agent. Despite requiring more human data, this enables to make no assumption about the form of human behavior (as it is learned from data). In this work, we try to make access to vast quantities of human data less of a bottleneck, both for quality of human models, and of AIs meant to collaborate with humans.

Generalization to new environments. Much work has been focused on creating agents that can perform well in a variety of environments, especially for the purposes of sim-to-real transfer [48]. One common technique is domain randomization, which involves defining a distribution over environments $\mathcal{E}$ and training the agent to perform well in any randomly drawn environment [47, 41, 19]. We use this approach for our agent training in order to have it generalize to new environments at test time.

Multi-agent MDPs. A $n$-player MDP $\mathcal{M}$ is defined by a tuple $\big{\langle}\mathcal{S},\{\mathcal{A}_{i}\}_{1:n},\mathcal{P},\mathcal{T},\gamma,\mathcal{R},H\big{\rangle}$. $\mathcal{S}$ is a finite set of states; $\mathcal{A}_{i}$ is the finite set of actions available to agent $i$; $\mathcal{P}$ is the initial state distribution; $\mathcal{T}:\mathcal{S}\times\mathcal{A}_{1}\times\dots\times\mathcal{A}_{n}\times\mathcal{S}\to[0,1]$ is a transition function which specifies the distribution over the next state, given all agents’ actions; $\gamma$ is the discount factor, and lastly $\mathcal{R}:\mathcal{S}\to\mathbb{R}$ is a real-valued reward function. We additionally denote the expected reward obtained by a policy tuple $\pi_{1},\dots,\pi_{n}$ by $\rho(\pi_{1},\dots,\pi_{n})=\mathbb{E}_{\pi_{1},\dots,\pi_{n}}[\sum^{H}_{t=0}r_{t}]$.

Embedding agents in multi-agent MDPs. If we are given every agents’ policy except for the $k$th policy $\{\pi_{i}:i\neq k\}$, then considering the multi-agent MDP $\mathcal{M}$ from the perspective of agent $k$, the other agents can be considered as “part of the transition dynamics of the environment”. The problem of finding the optimal $\pi_{k}$ of this multi-agent MDP $\mathcal{M}$ can thus be reduced to finding the optimal $\pi$ for a single-agent MDP $\dot{\mathcal{M}}$ in which the transition dynamics compose the choices of actions by the other agents $\{\pi_{i}:i\neq k\}$, and the original transition dynamics $\mathcal{T}$ [12].

Human-AI collaboration through deep RL. In the simplest human-AI collaboration setting, we are given a two-player MDP $\mathcal{M}$ consisting of a human and an AI agent. If the human policy $\pi_{H}$ were known, we could simply embed the policy in the environment (as described above) and use RL to obtain an optimal collaborative policy. Given that sampling from the true human policy $\pi_{H}$ is expensive, training a deep RL policy with the human fully in-the-loop is unfeasible. The approach taken by previous work [6, 29] is to leverage human data to learn a human model $\hat{\pi}_{H}$, embed it into the environment, and then use deep RL to find an optimal policy for this induced single-agent MDP $\dot{\mathcal{M}}$. If the human model is of sufficiently high quality (i.e. $\hat{\pi}_{H}$ is sufficiently close to $\pi_{H}$), the resulting agent policy will be able to collaborate with humans at test time.

Best Response (BR). In a two-player collaborative game, the best response operator $\text{BR}:\Pi\rightarrow\Pi$ takes in a fixed partner policy $\pi$ and returns an agent policy that maximizes the expected collaboration rewards $\rho$ when paired with $\pi$: $\text{BR}(\pi)=\operatorname*{arg\,max}_{\tilde{\pi}\in\Pi}\mathbb{E}[\rho(\tilde{\pi},\pi)]$. Note that computing the optimal policy for the single-agent MDP $\dot{\mathcal{M}}$ above is equivalent to computing the best response to the human model $\pi_{H}$. In environments that are sufficiently complex, computing best responses exactly is unfeasible, leading to the usage of approximate methods: in our setting, deep RL can be considered our choice of approximate best response operator $\widehat{\text{BR}}$ [12].

Setup. We want to obtain a human model of the highest possible quality given a limited set of human data that we have available. Let $\mathcal{E}$ be a distribution over the possible environments that we are interested in deploying our system in. We have access to human-human gameplay data for $N$ environments from the distribution $\mathcal{E}$. That is, we will have a dataset of a human collaborating with another human $D_{HH}^{i}$ with $i\in\{1,\dots,N\}$, for each environment $e_{i}\sim\mathcal{E}$. Note that when $\mathcal{E}$ deterministically returns a single-environment, this setup is equivalent to a more traditional single-environment case.

Optimal-Behavior Prior (OBP). We leverage the observation that – from a Bayesian perspective – the parameter initialization of a human model can be thought of as encoding a specific prior over the model space [23, 11]. The standard approach of randomly initializing parameters from high-capacity human models – insofar as inputs will tend to output high-entropy distributions over actions – is equivalent to a prior that humans will have highly random behavior. Our insight is that while human behavior is generally not optimal (or one could use optimal agents as human models), in the vast majority of tasks, it is a fair assumption to expect humans to be better at the task than random behavior (i.e. humans “try to be optimal”). This leads us to expect that substituting the standard “random behavior” prior with an “optimal behavior prior” would facilitate the approximate Bayesian inference enacted by the model optimization [23], potentially requiring significantly fewer iterations (and less data) to achieve the same level of human model quality. In practice, we show that even the simplest approach to encode such a prior – using the model weights from a trained (near)-optimal agent as weight initialization for our human model – works much better.

Computing OBP. Straightforwardly applying this idea requires the optimal behavior model (from which we intend to take the parameterization) and the human model to be of the same form. We consider neural network models, as they are plausibly expressive enough to represent both optimal and human behavior with the same model form. To obtain optimal behavior to use as prior for our multi-environment human model, we train a policy in self-play over the distribution of environments $\mathcal{E}$, leading to a near-optimal agent over $\mathcal{E}$, $\text{SP}^{\mathcal{E}}$, whose parameterization we can take and use to warm-start the training of our human model $\text{BC}^{\mathcal{E}}_{\text{OBP}}$ (see Figure 1 and Algorithm 1).

OBP with freezing ($\text{OBP}+f$). One can also view OBP through the lens of representation learning [5]: the representations learned by optimal agents will likely be more relevant to the task than randomly initialized ones. We investigate whether the intermediate representations learned by optimal agents are sufficient to model human behavior, by freezing the first $k$ network layers from the OBP initialization and only fine-tuning the remaining network layers. From the Bayesian perspective, this can be seen as extra regularization, by encoding a Dirac prior on certain components of the model (i.e. complete certainty about the representations up until layer $k$) – this assumption of absolute certainty about representations is thus implicitly equivalent to the assumption that the remaining free parameters still contain enough expressivity to represent human behavior.

Overview of assumptions. To summarize, our main assumptions are: access to a distribution of environments of interest $\mathcal{E}$, similar to ones we want to be able to collaborate with humans in at test time; access to human dataset from a limited number of diverse environments $D_{HH}^{1:N}$; being able to train (near-)optimal behavior over environments in $\mathcal{E}$; being able to use the same model form for our human model as the (near-)optimal behavior parameterization. Finally, we are assuming that human behavior at the task at hand is closer to optimality than to chance.

Having obtained a general human model, we can then extend the deep RL method described in Section 3 to train collaborative AIs across the whole of $\mathcal{E}$ through domain randomization. See Algorithm 1 for the full algorithm for training the OBP human model and the collaborative AI agent.

Hypotheses. In our experiments, we hope to investigate the following two hypotheses:

H1. OBP enables human models to better generalize to unseen environments, measured by reward similarity to H and validation loss.

H2. When trained with a human model using OBP initialization, human-aware AI agents become better at collaborating with real humans on unseen environments, as measured by task reward.

To validate H1 and H2, we respectively consider a variety of possible training regimes for human models (Section 5.1), and collaborative agents (Section 5.2).

Metrics for evaluating human models. One takeaway from our experiments is that it’s not obvious how one should judge the quality of human models. Human models can have the exact same loss (for human data prediction) but lead to radically different qualitative behavior. Similarity in reward to humans seems like a potentially better metric, but it’s also not bulletproof: in the most extreme case, memorizing the training data would lead to the exact same reward as the demonstrations.

Limitations and future work. Firstly, while we performed our experiments with simulated ground truth proxy humans, more work is required to verify that these results will hold with real humans. In addition, a clear avenue of future work would be that of exploring other behavior priors which might be closer to human than optimal behavior, including but not limited to max-entropy agents. Another interesting avenue of future work regards the warm-starting of the training of optimal agents: in complex domains, it’s common to bootstrap RL with a imitation learning policy to speed up training [28, 45]. Inspired by this, we speculate whether it may be advantageous to combine these approaches, first training a simple BC model (which is very fast) which can be used to bootstrap RL training (as it helps address exploration problems), and then – assuming that the information in the human data would be lost to catastrophic forgetting – one could using the resulting RL policy to boostrap the training of a better human model (via OBP).

Summary. In our work, we try to create generalizable human models for the purposes of improving human modeling and human-AI collaboration. While straightforward multi-environment human modeling struggles to learn anything meaningful, we find that good quality multi-environment human modeling is rendered possible by leveraging the knowledge that humans will be better than random with an optimal behavior prior. We then show that this type of modeling approach is also advantageous for downstream collaborative reward, performing better than modeling the human as optimal.

We’d like to thank members of the InterACT lab for helpful discussion. This work was supported by NSF CAREER, NSF NRI, ONR YIP, and the NSF Fellowship.