Reward-Poisoning Attacks on Offline Multi-Agent Reinforcement Learning

We introduce reward-poisoning attacks in offline MARL. We show that any attack that reduces to attacking single-agent RL separately must be suboptimal. Consequently, new innovations are necessary to attack effectively. We present a reward-poisoning framework that guarantees the target policy $\pi^{\dagger}$ becomes a Markov Perfect Dominant Strategy Equilibrium (MPDSE) for the underlying Markov Game. Since any rational agent will follow an MPDSE if it exists, this ensures the agents adopt the target policy $\pi^{\dagger}$. We also show the attack can be efficiently constructed using a linear program.

The attack framework has several important features. First, it is effective against a large class of offline MARL learners rather than a specific learning algorithm. Second, the framework allows partially decentralized agents who can only access their own individual rewards rather than the joint reward vectors of all agents. Lastly, the framework only makes the minimal assumption on the rationality of the learners that they will not take dominated actions.

We also give interpretable bounds on the minimal cost to poison an arbitrary dataset. These bounds relate the minimal attack cost to the structure of the underlying Markov Game. Using these bounds, we derive classes of extremal games that are especially cheap or expensive for the attacker to poison. These results show which games may be more susceptible to an attacker, while also giving insight to the structure of multi-agent attacks.

In the right hands, our framework could be used by a benevolent entity to coordinate agents in a way that improves social welfare. However, a malicious attacker could exploit the framework to harm learners and only benefit themselves. Consequently, our work paves the way for future study of MARL defense algorithms.

We assume that the attacker has access to the original dataset $\mathcal{D}$. The attacker has a pre-specified target policy ${\boldsymbol{\pi}}^{\dagger}$ and attempts to poison the rewards in $\mathcal{D}$ with the goal of forcing the learners to learn ${\boldsymbol{\pi}}^{\dagger}$ from the poisoned dataset. The attacker also desires that the attack has minimal cost. We let $C(r^{0},r^{\dagger})$ denote the cost of a specific poisoning, where $r^{0}=\big{\{}\big{(}{\boldsymbol{r}}_{h}^{0,(k)}\big{)}_{h=1}^{H}\big{\}}_{k=1}^{K}$ are the original rewards and $r^{\dagger}=\big{\{}\big{(}{\boldsymbol{r}}_{h}^{\dagger,(k)}\big{)}_{h=1}^{H}\big{\}}_{k=1}^{K}$ are the poisoned rewards. We focus on the $L^{1}$-norm cost $C(r^{0},r^{\dagger})=\|r^{0}-r^{\dagger}\|_{1}$.

As a stepping stone, we start with a subclass of Markov Games with $|\mathcal{S}|=1$ and $H=1$, which are sometimes called bandit games. A bandit game consists of a single stage normal-form game. For now, we also pretend that the learners simply use the data to compute an MLE point estimate $\hat{G}$ of the game and then solve the estimated game $\hat{G}$. This is unrealistic, but it highlights the attacker’s strategy to enforce that ${\boldsymbol{\pi}}^{\dagger}$ is an $\iota$-strict DSE in $\hat{G}$.

Suppose the original dataset is $\mathcal{D}=\big{\{}({\boldsymbol{a}}^{(k)},{\boldsymbol{r}}^{0,(k)})\big{\}}_{k=1}^{K}$ (recall we no longer have state or period). Also, let $N({\boldsymbol{a}}):=\sum_{k=1}^{K}\mathbf{1}_{\{{\boldsymbol{a}}^{(k)}={\boldsymbol{a}}\}}$ be the action counts. The attacker’s problem can be formulated as a convex optimization problem given in (1).

The first constraint in (1) models the learners’ MLE $\hat{G}$ after poisoning. The second constraint enforces that ${\boldsymbol{\pi}}^{\dagger}$ is an $\iota$-strict DSE of $\hat{G}$ by definition. We observe that:

1. 1.

   The problem is feasible if $\iota\leq 2b$, since the attacker can always set, for each agent, the reward to be $b$ for the target action and $-b$ for all other actions;

2. 2.

   If the cost function $C(\cdot,\cdot)$ is the $L^{1}$-norm, the problem is a linear program (LP) with $nK$ variables and $(A-1)A^{n-1}+2nK$ inequality constraints (assuming each learner has $\left|\mathcal{A}_{i}\right|=A$ actions);

3. 3.

   After the attack, learner $i$ only needs to see its own rewards to be convinced that $\pi_{i}^{\dagger}$ is a dominant strategy; learner $i$ does not need to observe other learners’ rewards.

This simple formulation serves as an asymptotic approximation to the attack problem for confidence bound based learners. In particular, when $N({\boldsymbol{a}})$ is large for all ${\boldsymbol{a}}$, the confidence intervals on $P$ and $R$ are usually small.

With the above idea in place, we can consider more realistic learners that are uncertainty-aware. For these learners, the attacker attempts to enforce an $\iota$ separation between the lower bound of the target action’s reward and the upper bounds of all other actions’ rewards (similar to arm elimination in bandits). With such separation, all plausible games in $\textup{CI}^{G}$ would have the target action profile as the dominant strategy equilibrium. This approach can be formulated as a slightly more complex optimization problem (2), where the second and third constraints enforce the desired $\iota$ separation. The formulation (2) can be solved using standard optimization solvers, hence the optimal attack can be computed efficiently.

We next consider whether this formulation has a feasible solution. Below we characterize the feasibility of the attack in terms of the margin parameter $\iota$ and the confidence bounds.

Proposition 2 is a special case of the general Theorem 5 with $H=\left|\mathcal{S}\right|=1$. We note that the condition in Proposition 2 has an equivalent form that relates to the structure of the dataset. We later present this form for more general case.

When an $L^{1}$-norm cost function is used, we show in the appendix that the formulation (2) can also be efficiently solved.

We now generalize the ideas from the bandit setting to derive a poisoning framework for arbitrary Markov Games. With multiple states and periods, there are two main complications:

1. 1.

   In each period $h$, the learners’ decision depends on $Q_{h}$, which involves both the immediate reward $R_{h}$ and the future return $Q_{h+1}$;

2. 2.

   The uncertainty in $Q_{h}$ amplifies as it propagates backward in $h$.

Accordingly, the attacker needs to design the poisoning attack recursively.

Our main technical innovation is an attack formulation based on $Q$ confidence-bound backward induction. The attacker maintains confidence upper and lower bounds on the learners’ $Q$ function, $\overline{Q}$ and $\underline{Q}$, with backward induction. To ensure ${\boldsymbol{\pi}}^{\dagger}$ becomes an $\iota$-MPDSE, the attacker again attempts to $\iota$-separate the lower bound of the target action and the upper bound of all other actions, at all states and periods.

Recall Definition 2: given the training dataset $\mathcal{D}$, one can compute the MLEs ${\boldsymbol{R}}_{h}$ and corresponding confidence sets $\textup{CI}_{i,h}^{R}$ for the reward. The attacker aims to poison $\mathcal{D}$ into $\mathcal{D}^{\dagger}$ so that the MLEs and confidence sets become ${\boldsymbol{R}}_{h}^{\dagger}$ and $\textup{CI}_{i,h}^{R^{\dagger}}$, under which ${\boldsymbol{\pi}}^{\dagger}$ is the unique $\iota$-MPDSE for all plausible games in the corresponding confidence game set. The attacker finds the minimum cost way of doing so by solving a $Q$ confidence-bound backward induction optimization problem, given in (3)–(7).

The backward induction steps (4) and (5) ensure that $\underline{Q}$ and $\overline{Q}$ are valid lower and upper bounds for the $Q$ function for all plausible Markov Games in $\textup{CI}^{G}$, for all periods. The margin constraints (6) enforces an $\iota$-separation between the target action and other actions at all states and periods. We emphasize that the agents need not consider $Q$ at all in their learning algorithm; $Q$ only appears in the optimization due to its presence in the definition of MPDSE.

Again, pairing an efficient optimization solver with the above formulation gives an efficient algorithm for constructing the poisoning. We now answer the important questions of whether this formulation admits a feasible solution and whether these solutions yield successful attacks. The lemma below provides a positive answer to the second question.

Moreover, the attack formulation admits feasible solutions under mild conditions on the dataset.

We remark that the learners know the upper bound $b$ and may use it exclude implausible games. The accumulation of confidence intervals over the $H$ periods results in the extra factor $(H+1)$ on $\rho^{R}_{h}$. Theorem 5 implies that the problem is feasible so long as the dataset is sufficiently populated; that is, each $(s,a)$ pair should appear frequently enough to have a small confidence interval half-width $\rho^{R}_{h}$. The following corollary provides a precise condition on the visit accounts that guarantees feasibility.

Despite the inner min and max in the problem (3)–(7), the problem can be formulated as an LP, thanks to LP duality.

With the above definitions, we present universal attack cost bounds that hold simultaneously for all attack instances.

Specifically, consider the dataset for a bandit game, $\mathcal{D}=\big{\{}({\boldsymbol{a}}^{(k)},{\boldsymbol{r}}^{0,(k)})\big{\}}_{k=1}^{K},$ where $\mathcal{A}=A^{n}$ and each action appears exactly $N$ times, i.e., $\overline{N}=\underline{N}=N$ and $K=NA^{n}$. The target policy is ${\boldsymbol{\pi}}^{\dagger}=(1,\ldots,1)$. The dataset is constructed so that ${\boldsymbol{r}}^{0,(k)}_{i}=-b$ if ${\boldsymbol{a}}^{(k)}_{i}=\pi_{i,h}^{\dagger}\left(s\right)$ and ${\boldsymbol{r}}^{0,(k)}_{i}=b$ otherwise. These rewards are essentially the extreme opposite of what the attacker needs to ensure ${\boldsymbol{\pi}}^{\dagger}$ is an $\iota$-DSE. Note, the dataset induces the MLE of the game shown in Table 1 for the special case with $n=2$ players.

For simplicity, suppose that the same confidence half-width $\rho^{R}\left({\boldsymbol{a}}\right)=\rho<b$ is used for all ${\boldsymbol{a}}$. Let $\iota\in(0,b)$ be arbitrary. For this instance, to install ${\boldsymbol{\pi}}^{\dagger}$ as the $\iota$-DSE, the attacker can flip all rewards in a way that is illustrated in Table 2, inducing a cost as the upper bound in Theorem 8. The situation is the same for $n\geq 2$ learners.

Our instance-dependent lower bound, presented later in Theorem 12, implies that any attack on this instance must have cost at least $NnA^{n-1}(2b+2\rho+\iota)$. This lower bound matches the refined upper bound in the proof of Theorem 9, implying the refined bounds are tight for this instance. Noticing that the universal bound in Theorem 8 only differs by an $O(A)$-factor implies it is nearly tight.

Next, we derive general bounds on the attack cost that depend on the structure of the underlying instance. Our strategy is to reduce the problem of bounding Markov Game costs to the easier problem of bounding Bandit Game costs. We begin by showing that the cost of poisoning a Markov Game dataset can be bounded in terms of the cost of poisoning the datasets corresponding to its individual period games.

The lower bound is valid for all Markov Games, but it is weak in that it only uses the last period cost. However, this is the most general lower bound one can obtain without additional assumptions on the structure of the game. If we assume additional structure on the dataset, then the above lower bound can be extended beyond the last period, forcing a higher attack cost.

Now that we have the above relationships, we can focus on bounding the attack cost for bandit games. To be precise, we bound the cost of poisoning a period game instance $I_{h}$. To this end, we define $\iota$-dominance gaps.

The lower bounds in both Lemma 10 and Lemma 11 expose an exponential dependency on $n$, the number of players, for some datasets $\mathcal{D}$. These instances essentially require the attacker to modify $\hat{R}_{i,h}(s,{\boldsymbol{a}})$ for every ${\boldsymbol{a}}\in\mathcal{A}$. A concrete instance can be constructed by taking the high cost dataset derived as the tight example before and extend it into a general Markov Game. We simply do this by giving the game several identical states and uniform transitions. In terms of the dataset, each episode consists of independent plays of the same normal-form game, possibly with a different state observed. For this dataset the $\iota$-dominance gap can be shown to be $d_{i,h}^{\iota}\left(s,a_{-i}\right)=2b+2\rho+\iota$. A direct application of Lemma 10 gives the following explicit lower bound.

Recall the attacker wants to assume little about the learners, and therefore chooses to install an $\iota$-MPDSE (instead of making stronger assumptions on the learners and installing a Nash equilibrium or a non-Markov perfect equilibrium). On some datasets $\mathcal{D}$, the exponential poisoning cost is the price the attacker pays for this flexibility.