Modeling Non-Cooperative Dialogue: Theoretical and Empirical Insights

An image is randomly selected and an object within this image is randomly chosen to be the goal-object. With some probability, the game instance is designated as a cooperative game. Otherwise, the game is non-cooperative.

Unlike the original GuessWhat?! game, there are three (not two) player roles: the question-player, the cooperative answer-player, and the non-cooperative answer-player. For cooperative game instances (decided at initialization), the cooperative answer-player is put in play. Otherwise, the non-cooperative answer-player is put in play. The question-player always plays and does not know whether the answer-player is cooperative or non-cooperative. To start, all active players are granted access to the image. The question-player asks yes/no questions about the image and objects within the image. At the end of dialogue, the question-player will use the gathered information to guess both the unknown goal-object and the (cooperation) type of the active answer-player.³³3This is done simultaneously, so knowledge of the correctness of one guess cannot inform the other guess. Unlike the question-player, the active answer-player has knowledge of the game’s goal-object and responds to the question-player’s queries with yes, no, or n/a (not applicable).

The question-player’s goals are always to identify both the goal-object and the presence of non-cooperation if it exists (i.e., if the non-cooperative answer-player is in play). The cooperative answer-player’s goal is to reveal the goal-object to the question-player by answering the yes/no questions appropriately. The non-cooperative answer-player’s goal is instead to lead the question-player away from this goal object; i.e., to ensure the question-player does not correctly guess this object. There is no specific way in which this misleading must be done (e.g., there is not always an alternate object). Instead, during data collection, participants are simply instructed to deceive the question-player.

The question-player and active answer-player converse until the question-player is ready to make a guess or a pre-specified maxium number of dialogue rounds have transpired.⁴⁴4For data collection, no question limit is set. Experiments in Section 5 follow Strub et al. (2017) and set the max to 5. The question-player is then presented with a list of possible objects and must guess which of these was the secret goal-object. In addition, the question-player must guess whether the answer-player was cooperative or non-cooperative.

We developed a web application to collect dialogue from human participants taking the role of a non-cooperative answer-player. Participants were native English speakers recruited via an online crowd-sourcing platform and paid $15 per hour according to our institution’s human subject review board. Participants were asked to deceive an autonomous question-player pre-trained to identify the goal-object only. For pre-training, we used the original Guess What?! game corpus and supervised learning setup (De Vries et al., 2017; Strub et al., 2017). Participants received an image and a crop that indicated the goal-object. Both of these are randomly sampled from the original Guess What?! game corpus. They were tasked with leading the question-player away from this goal object by answering questions with yes, no, or n/a. Dialogue persisted until the question-player made a guess.

We collected 3746 non-cooperative dialogues. Dataset statistics are shown in Table 1, while visualization of the object and dialogue-length distributions are shown in Figure 2. Compared to the original Guess What?! corpus, both dialogue-length and object distributions are similar. For objects, this is expected as these are uniformly sampled from the original corpus. We see 16 of our 20 most likely objects are shared with the 20 most likely of the original GuessWhat?! object distribution, and further, the first 4 objects have identical ordering (see Figure 3). Differences here are simply attributed to randomness and the increasing uniformity as likelihood of an object decreases. For dialogue length, one might expect non-cooperative dialogue to be longer. Instead, the distributions are both right-skew with an average near 5 (i.e., 4.99 in our dataset and 5.11 in the original GuessWhat?! corpus). The primary difference is that the original corpus has more outliers, which is most probably a result of the increased sample size. We likely observe consistency between our non-cooperative corpus and the original corpus because the question-player – who controls dialogue length – is autonomous and trained on a cooperative corpus. Hence, this and other aspects of our non-cooperative corpus may be influenced by pre-conditioning the question-player for cooperation. This issue is mitigated in our experiments (Section 5) where the question-player is also trained on simulated non-cooperative dialogue. Also note, while the size of our collected dataset is smaller than the original cooperative corpus, we only use our data to train an autonomous, non-cooperative answer-player. When a larger sample is required (e.g., when training the question-player via RL), we use simulated non-cooperative data generated by the pre-trained, non-cooperative answer-player, which is a standard technique in the literature (Strub et al., 2017).

Besides the statistics shown in Table 1 and Figure 2, we also point out the question-player succeeded at identifying the goal-object in only 19% of the collected games. Comparatively, on an autonomously generated and fully cooperative test set, comparably trained question-players achieve 52.3% success (Strub et al., 2017). This indicates the deceptive strategies employed by the humans were effective at fooling the question-player to select the wrong goal-object. More detailed analysis of the strategies used by the participants is given in Section 5; these strategies are self-described by the participants and also automatically detected for a simple case. Finally, we also computed the answer distribution on the collected corpus: answers were 46% yes, 52% no, and 2% n/a.

We write $\mathcal{Y}$ to describe the finite set of object labels and $\mathcal{Z}=\{\mathtt{CP},\mathtt{NC}\}$ for the set of cooperation labels; $\mathtt{CP}$ denotes cooperation and $\mathtt{NC}$ denotes non-cooperation. In relation to the example in Figure 1, $\mathcal{Y}$ might contain labels for the orange, apple, cups, and dining-table. In the same example, the cooperation label would be $\mathtt{NC}$ to indicate a non-cooperative answer-player. We use $\mathcal{X}$ to denote the feature space which contains all possible game configurations. For example, each $X\in\mathcal{X}$ might capture the dialogue history, the image, and particular features of the image pre-extracted for the question-player (i.e., which objects are contained in the image at which locations). With this notation, the question-player’s learned hypotheses may be described as an object identification hypothesis $o:\mathcal{X}\to\mathcal{Y}$ and a cooperation identification hypothesis $c:\mathcal{X}\to\mathcal{Z}$. The question-player learns these functions by example. In particular, we assume the question-player is given access to a random sequence of $m$ examples $S=(X_{i},Y_{i},Z_{i})_{i=1}^{m}$ independently and identically distributed according to an unknown distribution $\mathbb{P}_{\theta}$ over $\mathcal{X}\times\mathcal{Y}\times\mathcal{Z}$. To abbreviate, we write $S\overset{\mathrm{iid}}{\sim}\mathbb{P}_{\theta}$ and assume all samples are of size $m$ for simplicity. The distribution $\mathbb{P}_{\theta}$ is dependent on the question-player’s communication policy $\pi_{\theta}$ which we assume is uniquely determined by the real-vector $\theta$. Later, this allows us to select communication strategies using common reinforcement learning algorithms.

We emphasize the dependence of $\mathbb{P}_{\theta}$ on $\pi_{\theta}$ distinguishes our setup from typical scenarios in learning theory. Besides learning the hypotheses $o$ and $c$, the question-player can also select the communication policy $\pi_{\theta}$. This policy implicitly dictates the distribution over which the question-player learns, and thus, can either improve or hurt the player’s chance at success. As in reality, neither we nor the learner have knowledge of the mechanism through which changes to the communication policy $\pi_{\theta}$ modify the distribution $\mathbb{P}_{\theta}$. Our only assumption is that changing $\pi_{\theta}$ does not modify the probability of cooperation. That is, there is a constant $\mathbf{p}_{\mathtt{NC}}\in(0,1)$ such that for all $\pi_{\theta}$

This agrees with the description in Section 3 where the game instance is designated cooperative or non-cooperative prior to dialogue. With a random sample $S$, an unbiased estimate for $\mathbf{p}_{\mathtt{NC}}$ is

where $\mathbf{1}$ is the indicator function.

To measure the quality of the question-player’s guesses, we report the observed error-rate on the sample $S=(X_{i},Y_{i},Z_{i})_{i=1}^{m}$. In particular, the empirical object-identification error for any hypothesis $o:\mathcal{X}\to\mathcal{Y}$ is defined

Similarly, the cooperation identification error for any hypothesis $c:\mathcal{X}\to\mathcal{Z}$ is defined

In some cases, we instead restrict the sample over which we compute the empirical object-identification error. Specifically, we restrict to cooperative game instances and write

where $S^{\prime}=((X_{i},Y_{i})\mid Z_{i}=\mathtt{CP})$ is the sample $S$ with each triple where $Z_{i}\neq\mathtt{CP}$ removed. The case $\widehat{\mathbf{oer}}_{S}(o\mid\mathtt{NC})$ is defined similarly. Based on these, we further define the cooperation gap

This gap describes observed change in (weighted) object-identification error induced by change in cooperation. We often expect $\Delta$ to be positive.⁵⁵5Delta is negative when the object-identification error is higher on cooperative examples than non-cooperative examples (for simplicity, this assumes $\hat{\mathbf{p}}_{S}=0.5$). In practice, we rarely expect cooperation to lead to worse performance.

Finally, recall $S\overset{\mathrm{iid}}{\sim}\mathbb{P}_{\theta}$ and $\mathbb{P}_{\theta}$ is unknown, so in practice, we can only report the observed error discussed above. Still, we are typically more interested in the true or expected error for future samples from $\mathbb{P}_{\theta}$. This quantity tells us how the question-player’s hypotheses generalize beyond the random samples we observe. Precisely, the expected cooperation-identification error of a hypothesis $c:\mathcal{X}\to\mathcal{Z}$ is defined

where $(X,Y,Z)\sim\mathbb{P}_{\theta}$. The true (or expected) object-identification error is similarly defined.

Notice, one sensible choice of $o$ and $o^{\prime}$ is to pick $o$ which minimizes the observed object-identification error and $o^{\prime}$ which maximizes $\Delta_{S}$; this produces the tightest bound on the expected cooperation-identification error. We leave these hypotheses unspecified because later we must make limiting assumptions on the properties of $o$ and $o^{\prime}$ (e.g., Prop. 4.1). Greater generality here makes our results more broadly applicable. Besides this, we also observe that $C$ goes to 0 as $m$ grows. Ultimately, we ignore $C$ in interpretation, but point out that bounds based on the VC-Dimension (as above) are notoriously loose for most $\mathbb{P}_{\theta}$. As we are primarily interested in these bounds for purpose of interpretation and algorithm design, this is a non-issue. On the other hand, if practically computable bounds are desired, other (more data-dependent) techniques may be fruitful; e.g., see Dziugaite and Roy (2017).

As noted, the question-player has some control over the distribution $\mathbb{P}_{\theta}$ through the communication policy $\pi_{\theta}$. So, Thm. 4.1 can be interpreted to motivate indirect mechanisms for controlling the cooperation-identification error $\mathbf{cer}_{\theta}(\hat{g})$. Specifically, with respect to $\widehat{\mathbf{oer}}_{S}$, we can infer that improving performance on the object identification task should implicitly improve performance on the separate task of identifying non-cooperation. The term $\Delta_{S}$ also offers insight. It suggests certain non-cooperative answer-players – whose actions induce a large reduction in performance as compared to the cooperative answer-player – are easy to identify. Stated more plainly, non-cooperative agents reveal themselves by their non-cooperation; this is true, in particular, when their behavior causes large performance drops. In Section 4.3, we formalize these concepts further.

Notice, the result assumes the hypotheses $o,o^{\prime}$ and policies $\pi_{\theta},\pi_{\theta^{*}}$ are fixed a priori to drawing $S,T$. Hence, the bound is only valid for test sets independent from training. Regardless, it is still useful for interpretation and this style of bound produces tighter guarantees than conventional learning-theoretic bounds; i.e., from both analytic and empirical perspectives, respectively (Shalev-Shwartz and Ben-David, 2014; Sicilia et al., 2021). Like Thm. 4.1, we also use two hypotheses $o,o^{\prime}\in\mathcal{O}$, but the result is easily specified to the one hypothesis case by taking $o=o^{\prime}$ (albeit, this may loosen the bound). In any case, the assumption is not unreasonable. A policy $\pi_{\theta^{*}}$ – optimized with respect to just one hypothesis $o$ – may also offer relative improvement for other hypotheses distinct from $o$. For greater certainty, the term $\delta$ in the probability can be made arbitrarily small provided a large enough sample. Sensibly, the term $\gamma$ indicates the probability is also proportional to how much better the communication mechanism $\pi_{\theta^{*}}$ is where “better” is given precise meaning by comparing population statistics for the objective $J(\cdot)$ via $\alpha$. At minimum, we require $\alpha>\epsilon$, but $\epsilon$ should be small for suitably effective answer-players anyway. Finally, we again, safely ignore $O(C)$ terms, which go to 0 as $m$ grows.

The takeaway from Prop. 4.1 is an unexpectedly sensible strategy for game success: the question-player focuses communication efforts only on identifying the goal-object. When the non-cooperative agent is effective, this communication strategy essentially reduces an upperbound on the true cooperation-identification error. All the while, this strategy very obviously assists the object-recognition task as well. We again note the implication that non-cooperative agents can reveal themselves by their non-cooperation. The question-player need not expend additional effort to uncover them by dialogue actions.

While Thm. 4.1 alludes the interpretation given above – since the object-identification error is shown to control cooperation identification error in part –, Prop. 4.1 distinguishes itself because it considers all terms in the upperbound (not just $\widehat{\mathbf{oer}}$). This subtlety is important. In particular, a priori, one cannot be certain that improving the object-identification error from $S$ to $T$ also improves the cooperation gap $\Delta$. Instead, it could be the case that $\Delta$ decreases and the overall bound on $\mathbf{cer}$ is worsened. Aptly, Prop. 4.1 isolates the circumstances (i.e., related to Def. 4.2), which ensure this adverse effect does not occur. It shows us, under reasonable assumptions, the communication strategy discussed in our interpretation controls the whole bound in Thm. 4.1 and not just some part. As noted, drawing inference from only a portion of the bound can have unexpected consequences. In fact, this is the topic of much recent work in analysis of learning algorithms (Johansson et al., 2019; Wu et al., 2019; Zhao et al., 2019; Sicilia et al., 2022).

It is also worthwhile to note that setting the reward as $\rho(X,Y,Z)=\mathbf{1}[o(X)=Y]$ is also an appropriate strategy in the distinct fully cooperative Guess What?! game. The authors of the original GuessWhat?! corpus propose this reward exactly in their follow-up work Strub et al. (2017) which uses RL to learn communication strategies in the fully cooperative setting. Thus, the theoretical results of this section are exceedingly practical. They suggest, for effective non-cooperative agents, we may sensibly employ the same techniques in both the fully cooperative setting and the partially non-cooperative setting. This is beneficial, because the nature of our problem anticipates we will not know the setting in which we operate.

As a final note, we remark on how this result may be applied to properly motivate a reward which, a priori, can only be heuristically justified. Specifically, a very reasonable suggestion would be to combine the rewards in Eq. (11) and Eq. (12) via convex sum. Prior to our theoretical analyses, it is unclear that the two strategies would be complementary. Instead, the objectives could be competing, and so, this mixed strategy could lead to sub-par performance on both tasks. In light of this, our theoretical results help to understand this heuristic more formally. They suggest the two strategies are, in fact, complementary and outline the assumptions necessary for this to be the case. In contrast, empirical analyses can be much more specific to the data used, among other factors. This, in general, is a key differentiation between the analysis we have provided here and the oft-used appeal to heuristics.

We first give a Lemma.

Now, we proceed with the proof of Prop. 4.1.

The question-player consists of: a hypothesis $o$ which predicts the goal-object given the object categories, object locations, and the dialogue-history; a hypothesis $c$ which predicts cooperation given the same information; and the communication policy $\pi_{\theta}$ which generates dialogue given the image⁶⁶6The image is processed by a VGG network and these features initialize the LSTM state in Figure 4. and the current dialogue-history. Each is modelled by a neural-network. Architectures of $o$ and the policy $\pi_{\theta}$ are identical to the guesser model and questioner model described by Strub et al. (2017). We give an overview of the architectures in Figure 4 as well.

The cooperative answer-player is modeled by a neural-network with binary output dependent only on the goal-object and the most immediate question. Strub et al. (2017) demonstrate – in the cooperative case – that additional features do not improve performance. On the other hand, non-cooperative behaviors may require more complex modeling. We explore different features for the network modeling the non-cooperative answer-player. During experimentation, we condition on various combinations of the full (and immediate) dialogue-history, the image, and the goal-object. The architectures in both cases are based on the oracle model described by Strub et al. (2017) with the addition of an LSTM that allows conditioning on the full dialogue-history. See Figure 4 for an overview.

As noted, $o$ is assumed fixed before considering the task of $c$. In practice, we achieve this through supervised learning (SL) by training $o$ on human games in the Guess What?! (GW) corpus. Similarly, the cooperative answer-player is trained via SL on the GW corpus. The non-cooperative answer-player uses our novel corpus of non-cooperative games (see Section 3). Following Strub et al. (2017), we pre-train the communication policy $\pi_{\theta}$ using SL on the GW corpus. In some cases, $\pi_{\theta}$ is then taught a specific communication strategy by fine-tuning with RL on simulated dialogue. Dialogue is simulated by randomly sampling $Z\sim\mathrm{Bernoulli}(\mathbf{p}_{\mathtt{NC}})$, drawing an image-object pair uniformly at random from the GW corpus, and allowing the current policy $\pi_{\theta}$ and the already trained answer-player indicated by $Z$ to converse 5 rounds. The hypothesis $c$ is trained simultaneously on simulated dialogue during the RL phase of $\pi_{\theta}$ via SL. We do so because $c$ is assumed to minimize sample error in Thm 4.1. While simultaneous gradient methods only approximate this goal, it is more in line with assumptions than fixing $c$ a priori. In general, hyper-parameters are fixed for all experiments and are detailed in the code, which is publicly available. When possible, we follow the parameter choices of Strub et al. (2017). As an exception, we shorten the number of epochs in the RL phase to 10. Recall, the new network $c$ is trained in this phase as well. For $c$, the learning rate is 1e-4. The new non-cooperative answer-players are trained similarly to the cooperative answer-players – i.e., as in Strub et al. (2017) – but we remove early-stopping to avoid the need for a validation set. Our non-cooperative corpus is thus used in its entirety for training since all trained agents are evaluated on novel generated dialogue (see Section 5.2). When training with the GW corpus, we use the original train/val split.

Despite some slight deviations from the original Guess What?! training setup, we point out that our fully cooperative results are fairly similar. In Figure 5, we show error-rate on simulated, cooperative, test dialogues for our question-player trained solely on object-identification; the precise error-rate is 48.8%. For the most similar training and testing setup used by Strub et al. (2017), the question-player achieves an error-rate of 46.7%.

Between qualitative analysis of this data and conversations with the workers, we determined three primary human strategies for deception: spamming, absolute contradiction, and alternate goal objects. When spamming, participants would answer every question with the same answer; e.g., always answering no. Absolute contradiction was when participants determined the correct answer to the question-player’s query and then provided the negation of this. Finally, alternate goal objects describes the strategy of selecting an incorrect object in the image and providing answers as if this object was the correct goal. Of these, spamming is fairly easy to automatically detect; i.e., by searching for games where all answers are identical. We find 19% of the collected non-cooperative dialogues contain entirely spam answers. This, of course, does not account for mixed strategies within a game, but it does indicate the dataset is not dominated by the least complex strategy. Lastly, we remind the reader, some non-cooperative strategies directly describe violations of Gricean maxims. In particular, absolute contradiction and alternate goal objects violate the maxim of quality, while spamming violates the maxim of relevance. Due to the answer-player’s simple vocabulary and the greater control given to the question-player (i.e., in directing conversation topic and length), the maxims of manner and quantity are difficult for the answer-player to violate. So, it is expected observed strategies do not violate these maxims.

We further studied strategies in the autonomous non-cooperative answer-players. Notice, besides spamming, the human strategies may require knowledge of the full dialogue history as well as other objects in the image. We tested whether the autonomous answer-player utilized this information by training multiple answer-players with different information access: the first produced answers conditioned only on the goal-object and the most immediate question (1), the next two were also conditioned on the full dialogue-history (2) or the full image (3), and the last was conditioned on all of these features (4). We paired these non-cooperative answer-players with a question-player whose communication strategy focused on the object-identification task; i.e., using Eqn. (12). Answer-players 2, 3, and 4 induced an object-identification error outside a 95% confidence interval⁷⁷7An upper bound on true error induced by the 1st answer-player is 0.749 with confidence 95% (Hoeffding Bound $\approx$ 10K samples). The sample error of the 2nd, 3rd, and 4th answer-player are, respectively, 0.756, 0.757, and 0.752. of answer-player 1. In contrast, Strub et al. (2017) found that cooperative answer-players only needed access to the goal-object and the most immediate question to perform well. This result indicates the complexities inherent to deception and suggests that distinct strategies were learned when non-cooperative answer-players had access to more information. In the remainder, we focus on non-cooperative answer-player 2 with access to the full dialogue history. Our interpretation for answer-players 1, 3, and 4 is largely similar.

Our next observation concerns the formal definition of effective given in Section 4 Def. 4.2. While the limiting property required by the definition is not easy to measure empirically, we observe in Figure 5 that the object-identification error on non-cooperative examples is relatively stable across question-player communication strategies. This fact – that the non-cooperative answer-player exhibits behavior consistent with an effective answer-player – points to the validity of our theory. Recall, an effective answer-player is assumed in Prop. 4.1.

Finally, the primary conclusion of our theoretical analysis was that communication strategies which focus only on the object-identification task should be effective for both object-identification and cooperation-identification. Figure 5 confirms this. Selecting a communication strategy based on improving object-identification improves object-identification as expected. Further, on the potentially opposing objective of identifying non-cooperation, this strategy is also effective. It far improves over a random baseline and also improves over the baseline which uses no RL-based strategy. On the other hand, the communication strategy which focuses only on the identification of non-cooperation fails at the opposing task of object-identification. This strategy performs almost as badly as a random baseline when the percent of non-cooperative examples is large and is also consistently worse than the baseline which uses no RL. The mixture of both strategies seems to achieve good middle ground. Recall, while this strategy may be heuristically intuited, our theoretical results formally justified this strategy as well.