Data Injection Attacks on Smart Grids with Multiple Adversaries: A Game-Theoretic Perspective

Competitive energy markets’ architectures are often based on day ahead (DA) and real time (RT) markets [8]. In the DA market, the system operator issues hourly-based locational marginal prices (LMPs), ${\mu^{DA}}$, for the next operating day based on the DA energy bids submitted by the participants [8]. The market clearing process is performed by the grid operator through the solution of a linearize Optimal Power Flow (DCOPF) which returns the optimal dispatch for each of the generators participating in the market and the DA LMP at each bus. The most commonly used DCOPF formulation is as follows [8]:

where $N,G$ and $L$ represent, respectively, the number of buses, generators, and transmission lines. $C_{i}$ corresponds to the offer of generator $i$ while $P_{i}$ and $D_{i}$ are, respectively, the power injection and load at a bus $i$. Thus, $P_{i}=0$ ($D_{i}=0$) corresponds to the case in which no generator (or load) is connected to bus $i$. The upper and lower limits on generator $i$’s output are denoted by $P_{i}^{\textrm{min}}$ and $P_{i}^{\textrm{max}}$. Constraints (4) and (5) place a limit, $F_{l}^{\textrm{max}}$, on the level of power that can flow over line $l$. A reference direction is assigned to the power flow over each transmission line. In this regard, a power flow opposing its assigned reference direction is represented by a negative quantity. Hence, constraints (4) and (5) correspond to the thermal limit of a line in its reference and opposite directions respectively. $\boldsymbol{X}$ is the generation shift factor matrix which defines the sensitivity of the power flow over each line, ${\boldsymbol{F}}$, to changes in power injection, $\boldsymbol{P}$, at each bus:

Therefore the sensitivity of the flow over line $l$ to a change in power injection at bus $i$ is denoted by $\chi_{l,i}$.

In the RT market, actual real-time operating conditions estimated using the state estimator, in lieu of the predictions in DA, are used through an ex-post model to compute the RT LMPs, ${\mu^{RT}}$  [8]. An incremental DCOPF is used to compute the RT LMPs and can be formulated as follows [8]:

where $C_{i}^{RT}$ is the RT offer of generator $i$ which is computed using its RT power output and its associated offer curve [8]. $\mathcal{C^{+}}$ ($\mathcal{C^{-}}$) is the set of congested lines which flow is in (opposite to) their reference directions. $\Delta P_{i}^{\textrm{max}}$ and $\Delta P_{i}^{\textrm{min}}$ define a bandwidth which is employed to allow for solution tolerance. In practice, here, we typically [9] set $\Delta P_{i}^{\textrm{min}}=-2$ MW and $\Delta P_{i}^{\textrm{max}}=+0.1$ MW. A proposed alternative to using this feasibility bandwidth is also available in [9].

Thus, the DA and RT LMPs at each bus, $i$, are computed using the DA and ex-post DCOPFs. The generated LMPs reflect, both, the incremental cost of energy at bus $i$ and the congestion cost associated with the contribution of this bus to the system congestion. A line is said to be congested if the flow of power over it reaches its maximum limit. The DA and RT LMPs at bus $i$ are given by:

$\mathcal{C}_{l}\triangleq\{\mathcal{C^{+}}\cup\mathcal{C^{-}}\}$ is the set of congested lines, in RT, obtained using the state estimator. $\mathcal{C}_{l}\subseteq\mathcal{L}$ where $\mathcal{L}=\{1,\dotsb,L\}$ is the set of all lines. $\lambda_{0}$ corresponds to the Lagrange multiplier associated with the energy balance constraints (2) and (8). $\lambda_{l}^{DA,+}$ and $\lambda_{l}^{DA,-}$ are the Lagrange multipliers corresponding, respectively, to constraints (4) and (5) for line $l\in\mathcal{L}$ whereas $\lambda_{l}^{RT,+}$ and $\lambda_{l}^{RT,-}$ are the Lagrange multipliers corresponding, respectively, to constraints (10) and (11) for line $l\in\mathcal{C}_{l}$. When $l\in\mathcal{L}$ but $l\notin\mathcal{C}_{l}$, $\lambda_{l}^{RT,+}=\lambda_{l}^{RT,-}=0$. Moreover, when $l\in\mathcal{C}^{+}$, $\lambda_{l}^{RT,-}=0$; while when $l\in\mathcal{C}^{-},\lambda_{l}^{RT,+}=0$.

Computing the RT LMPs relies on the ex-post DCOPF formulation which depends on the output of the state estimator. Hence, data injection attacks targeting the state estimation affects the LMPs in (13). Next, we introduce the data attack model.

A power system state estimator uses multiple power measurements collected throughout the grid to estimate the system states [10]. The relation between the measurement vector, ${\boldsymbol{z}}$, and the vector of system states, $\boldsymbol{\theta}$, in a linearized state estimation model (DC SE) is expressed as follows:

where $\boldsymbol{H}$ is the measurement Jacobian matrix and ${\boldsymbol{e}}$ is the vector of random errors assumed to follow a normal distribution, ${N(0,\boldsymbol{R})}$. Using a weighted least square (WLS) estimator the estimated system states are given by [10]:

Using the estimated states, an estimate of the measurement vector, $\hat{\boldsymbol{z}}$, and residuals, $\boldsymbol{r}$, can be calculated as follows [10]:

where ${I}_{n}$ is the identity matrix of size $(n\times n)$, and $n$ is the total number of collected measurements.

When data injection attacks are concurrently carried out by ${M}$ attackers in the set $\mathcal{M}=\{1,\ldots,M\}$, the collected measurements are modified through the addition of their corresponding attack vectors denoted by ${\{\boldsymbol{z}^{(1)},\boldsymbol{z}^{(2)},...,\boldsymbol{z}^{(M)}\}}$ resulting in the following altered measurements and residuals:

In the case in which the measurement errors $\boldsymbol{e}$ follow a normal distribution, the WLS estimator is a maximum likelihood estimator of location of the system states [10]. However, the WLS estimator has a zero robustness against outliers. To overcome this drawback, outliers’ detection and identification mechanisms are used so that the final state estimate is only based on “good data”. The measurement residuals give an indication of the real and unknown measurement errors. By replacing the expression of $\boldsymbol{z}$ from (14) in the expression of $\boldsymbol{r}$ in (16), the residuals can be expressed in terms of the true errors as follows [10]:

Thus, an analysis of the residuals allow for the detection and identification of bad data (outliers). In this respect, bad data detection corresponds to determining whether the collected measurement set contains bad data or not. On the other hand, bad data identification corresponds to identifying which measurements may contain bad data. One should note here that outliers can stem from data injection as well as other reasons such as meter biases or communication link failures [10].

Bad data detection is typically performed using a test known as the Chi-squares test over the sum of the squares of the residuals [1, 10]. In fact, when the measurement errors vector $\boldsymbol{e}$ is assumed to follow a normal distribution, $||\boldsymbol{r}||_{2}^{2}=\sum_{i=1}^{n}r_{i}^{2}$ follows a $\chi^{2}$ distribution with $n-N_{\theta}$ degrees of freedom where $N_{\theta}$ is the number of states to be estimated [10]. Hence, for a measurement set to be considered free from bad data, the residuals must satisfy $||\boldsymbol{r}||_{2}\leq\tau$ where $\tau$ is a detection threshold [1, 10]. In this respect, in the presence of $M$ attackers, and since $||\boldsymbol{r}^{\textrm{att}}||_{2}=||\boldsymbol{r}+\boldsymbol{W}\sum_{m=1}^{M}\boldsymbol{z}^{(m)}||_{2}$ as shown in (17), each attacker $m\in\mathcal{M}$ should regulate $\boldsymbol{W}\boldsymbol{z}^{(m)}$ to keep the effect of the attacks on the residuals low to minimize the chance of being detected as outliers [1].

In the case where the Chi-squares test indicates the presence of bad data, various bad data identification and elimination tests can be employed such as the largest normalized residual test, or the hypothesis testing identification (HTI), among others, to identify and eliminate the outliers [10].

In our model, each attacker $m\in\mathcal{M}$ aims at manipulating RT LMPs, ${\mu^{RT}}$, to make financial benefit via virtual bidding. Using virtual bidding, entities that do not own any physical generation nor load can engage in the energy market settlements by submitting so-called virtual supply and demand offers. Since these energy offers are virtual, an entity offering to buy (sell) virtual power at a given bus in DA is required to sell (buy) that same amount of power at the same bus in RT. Using such virtual bids, the grid operator aims at promoting liquidity in the energy market while, on the other hand, virtual bidders aim to reap financial profit from possible mismatch between the DA and RT LMPs [8]. Using a data injection attack, a virtual bidder can, thus, manipulate the RT LMPs to create a lucrative mismatch with respect to their DA counterparts. On the other hand, to achieve pricing integrity, the system operator aims at protecting the system against such attacks. The strategic interactions between the attackers and the defender (i.e. system operator) are modeled and analyzed next.

We formulate a strategic noncooperative game to analyze the optimal decision making of the $M$ attackers in response to any arbitrary defender strategy. This game is formulated in its normal form as follows: $\Xi=\langle\mathcal{M},(\mathcal{Z}^{(i)})_{i\in\mathcal{M}},(U_{i})_{i\in\mathcal{M}}\rangle$, where $\mathcal{M}$ is the set of $M$ attackers, $\mathcal{Z}^{(i)}$ is the set of actions (attack vectors $\boldsymbol{z}^{(i)}\in\mathcal{Z}^{(i)}$) available to attacker $i\in\mathcal{M}$, and $U_{i}$ is the utility function of attacker $i$. Thus, each attacker, $m\in\mathcal{M}$, selects an attack vector, ${\boldsymbol{z}^{(m)}\in\mathcal{Z}^{(m)}}$ that maximizes its utility $U_{m}$. Let $\mathcal{K}_{m}$ denote the subset of measurements that $m$ can attack. Then, $\mathcal{Z}^{(m)}$ can be represented by a column vector with elements equal to 0 except for those in $\mathcal{K}_{m}$ which can take values within a compact range reflecting the range of magnitude of the attack.

The utility function of each attacker reflects the financial benefit obtained by virtual bidding. Using virtual bidding, each attacker $m$ buys and sells ${P_{m}}$ MW at, respectively, buses ${i_{m}}$ and ${j_{m}}$ in DA while, conversely in RT, attacker $m$ sells and buys ${P_{m}}$ MW at, respectively, buses ${i_{m}}$ and ${j_{m}}$. Thus, the goal of attacker $m\in{\mathcal{M}}$ is to optimize the following (Problem 1):

where ${c_{m}(\boldsymbol{z}^{(m)})}$ is the cost of attack, and ${\boldsymbol{z}^{-(m)}}$ denotes the strategy vector of all players except $m$. The number of measurements that can be attacked concurrently by $m$ as well as the attack levels (the level of modification of a measurement) are limited by $\mathcal{Z}^{(m)}$. Since $||\boldsymbol{r}^{\textrm{att}}||_{2}=||\boldsymbol{r}+\boldsymbol{W}\sum_{m=1}^{M}\boldsymbol{z}^{(m)}||_{2}\leq||\boldsymbol{r}||_{2}+||\boldsymbol{W}\boldsymbol{z}^{(m)}||_{2}+\sum_{l=1,l\neq m}^{M}||\boldsymbol{W}\boldsymbol{z}^{(l)}||_{2}$, $m\in\mathcal{M}$ chooses $\boldsymbol{z}^{(m)}$ as in (20), where $\epsilon_{m}$ is a chosen threshold, to minimize the chance of the attack of being detected as outliers.

In fact, considering the case of two attackers in which attacker $1$’s (attacker $2$’s) aim is to increase the estimated flow, $\hat{z}_{i}$ ($\hat{z}_{j}$), over a line $l_{i}$ $(l_{j})$ in order to create a false congestion. The objective of attacker $1$ (attacker $2$) is, hence, to achieve $\Delta\hat{z}_{i}\geqslant F^{\textrm{max}}_{l_{i}}-\hat{z}_{i}$ $(\Delta\hat{z}_{j}\geqslant F^{\textrm{max}}_{l_{j}}-\hat{z}_{j})$. Following from (21), the change introduced to $\hat{z}_{i}$ and $\hat{z}_{j}$ by the two attacks is stated as follows:

where $s_{i,j}$ denotes element $(i,j)$ of matrix $\boldsymbol{S}$. When the measurement errors are independent and identically distributed (i.e. $\boldsymbol{R}=\sigma^{2}\boldsymbol{I}_{n}$), $\boldsymbol{S}$ is a symmetric matrix. This property can be proven based on (15) and (16) through showing that $\boldsymbol{S}^{T}=\boldsymbol{S}$ when $\boldsymbol{R}=\sigma^{2}\boldsymbol{I}_{n}$. Since $\boldsymbol{S}$ is symmetric, ${s}_{i,j}={s}_{j,i}$. In the event where $s_{i,j}<0$, both attackers’ actions attenuate the effect of one another. In fact, since ${s}_{i,j}<0$, $z^{(2)}_{j}$ ($z^{(1)}_{i}$) reduces $\Delta\hat{z}_{i}$ ($\Delta\hat{z}_{j}$) preventing it from causing any congestion over line $l_{i}$ ($l_{j}$). In the contrary, if $s_{ij}>0$, each of the attackers’ actions would assist the other in achieving its objective. This result can be trivially generalized to the case of $M$ attackers.

Moreover, the payoffs of the different attackers (i.e. virtual bidders) are also significantly interdependent. In fact, as shown next, an attacker can collect financial benefit or endure loses due to the strategies played by other attackers.

In this regard, following from (III-A), attacker $m$’s payoff in the presence of $M$ attackers is governed by:

Replacing the expressions of the DA and RT LMPs from (12) and (13) in (24) yields:

As a result, following the sign of $(\chi_{l,j_{m}}-\chi_{l,i_{m}})$, determined by the choice of virtual bid nodes $i_{m}$ and $j_{m}$, an attack modifying the congestion status of a line $l$ between DA and RT can introduce a positive or negative payoff to attacker $m$.

The attackers’ payoff in (III-A) is a function of DA and RT LMPs. These LMPs are indirectly controlled by the attack vector, $\boldsymbol{z}^{(m)}$, which can control the existence of congestion over transmission lines and hence eventually affect the LMPs in (12) and (13). Thus, (III-A) can be rewritten using (12) and (13) as:

where $\zeta_{m}$ is given by (25). By dropping $P_{m}$ for being a constant, the objective of attacker $m$ is hence to

We define the two sets of lines $\mathcal{L}_{m}^{+}$ and $\mathcal{L}_{m}^{-}$ such that $\mathcal{L}_{m}^{+}=\{l\in\mathcal{L}|\chi_{l,j_{m}}-\chi_{l,i_{m}}>0\}$ and $\mathcal{L}_{m}^{-}=\{l\in\mathcal{L}|\chi_{l,j_{m}}-\chi_{l,i_{m}}<0\}$. Moreover, let $\mathcal{L}^{R}$ and $\mathcal{L}^{O}$, such that $\mathcal{L}=\{\mathcal{L}^{R}\cup\mathcal{L}^{O}\}$, be the sets of lines over which the power flows in, respectively, the reference and opposite to reference directions.

Attacker $m$ seeks to congest or decongest lines in a way that maximizes (27). In this regard, for a line $l\in\mathcal{L}_{m}^{+}$, i.e. $\chi_{l,j_{m}}-\chi_{l,i_{m}}>0$, attacker $m$ profits from creating a congestion over $l$ in the reference direction, causing $\lambda_{l}^{RT,+}$ to be positive. Thus, $m$ aims at creating a congestion over a line $l\in{\{\mathcal{L}_{m}^{+}\cap\mathcal{L}^{R}\}}$. Similarly, for $l\in\mathcal{L}_{m}^{-}$, $m$ benefits from causing a congestion over line $l$ in the direction opposite to its reference direction, causing $\lambda_{l}^{RT,-}$ to be positive. Accordingly, $m$ aims a creating a congestion over a line $l\in{\{\mathcal{L}_{m}^{-}\cap\mathcal{L}^{O}\}}$. Combining these two observations, attacker $m$ aims at creating congestions over lines $l\in\{(\mathcal{L}_{m}^{+}\cap\mathcal{L}^{R})\cup(\mathcal{L}_{m}^{-}\cap\mathcal{L}^{O})\}$. In a similar manner, an attacker would also seek to remove congestion from a line $l$ in order to set its $\lambda_{l}^{RT,+}$ or $\lambda_{l}^{RT,-}$ to zero in a way that maximizes (27). To this end, $m$ aims at removing congestions from lines $l\in\{(\mathcal{L}_{m}^{+}\cap\mathcal{L}^{O})\cup(\mathcal{L}_{m}^{-}\cap\mathcal{L}^{R})\}$.

Given that $\hat{\boldsymbol{F}}^{\textrm{att}}$ is a vector of random variables, attacker $m$ aims at altering the expected value of $\hat{\boldsymbol{F}}^{\textrm{att}}$ to achieve, with highest possible probability, the intended congestion creation or removal to maximize (26). In other words, to create (or remove) a congestion over a line $l$, attacker $m$ designs its attack so that $\mathbb{E}[\hat{F}_{l}^{\textrm{att}}]\geqslant F_{l}^{\textrm{max}}+\delta_{m}$ (or $\mathbb{E}[\hat{F}_{l}^{\textrm{att}}]\leqslant F_{l}^{\textrm{max}}-\delta_{m}$) and aims at maximizing this $\delta_{m}$ to increase its chances for achieving its congestion. Thus, attacker $m$ aims to solve the following optimization problem where $\boldsymbol{S}^{F}\triangleq\boldsymbol{H}_{F}\boldsymbol{M}$ (Problem 2):

where $\boldsymbol{z}^{(m)}\in\mathcal{Z}^{(m)}$. The constraints in (37) put some limits on the variables $\delta_{k_{m}}$ and $\alpha_{k_{m}}$ relative to the corresponding flow limit $F^{\textrm{max}}_{k_{m}}$ where $\beta$ and $\beta^{\prime}$ correspond to the fraction of $F^{\textrm{max}}_{k_{m}}$ that $\delta_{k_{m}}$ and $\alpha_{k_{m}}$ can take respectively. Thus, attacker $m$ aims at maximizing $\delta_{k_{m}}$ to increase the chance of creating congestions over lines $k_{m}\in\{(\mathcal{L}_{m}^{+}\cap\mathcal{L}^{R})\cup(\mathcal{L}_{m}^{-}\cap\mathcal{L}^{O})\}$ and removing congestions from lines $k_{m}\in\{(\mathcal{L}_{m}^{+}\cap\mathcal{L}^{O})\cup(\mathcal{L}_{m}^{-}\cap\mathcal{L}^{R})\}$.

On the other hand, due to resource limitation, an attacker cannot concurrently achieve all its favorable congestions. Thus, the $\alpha_{k_{m}}$ variables are relaxation variables to ensure feasibility of the optimization problem. However, this relaxation is accompanied with a penalty factor, $\gamma$, present in the objective function which reflects a decrease in the objective function of the attacker for the case in which a beneficial congestion creation or removal is not performed. Hence, when such a congestion manipulation is feasible, this penalty factor ensures that the attacker has a high incentive to perform this congestion manipulation.

Moreover, similarly to  (III-A), $c_{m}(\boldsymbol{z}^{(m)})$ is the cost associated with the attack. This cost function can be represented as a scaled norm of the attack vector, where $\kappa_{m}$ is the scaling factor and $m_{l}$ is the length of vector $\boldsymbol{z}^{(m)}$:

Following this formulation, one can see that the constraints of the optimization problems of each of the attackers are coupled. In other words, the strategy space of each attacker depends on the strategies selected by the other attackers. Games in which the constraints of the different players are coupled are known as generalized Nash equilibrium problems (GNEP). A widely used solution concept of these games is known as the generalized Nash equilibrium (GNE) which is defined as follows[13]:

We next prove the existence of a GNE for the attackers’ game.

The solution to GNEP problems can be obtained using a number of widely adopted solution concepts that are available in literature [15, 13, 16] where the applicability of each technique depends on the characteristics of the utility functions and action spaces. Given the strict concavity of the utility function of each attacker’s problem and the convexity of the action space, such techniques converge to a GNE for our derived formulation.

Under a given equilibrium of the followers, the leader (grid operator) selects a defense vector ${\boldsymbol{a}_{0}}$ that determines which measurements are to be made secure and able to block potential attacks. The objective of the defender is to minimize a cost function capturing the variation between the DA and RT LMPs, on all $N$ buses in the system, as follows:

where ${c_{0}({\boldsymbol{a}_{0}})}$ is the cost of defense, $P_{L}$ is the total system load and ${B_{0}}$ is the limit on the number of measurements that the operator can defend simultaneously. In (40), $\mu_{i}^{RT}$ depends on the strategies taken by the defender, $\boldsymbol{a}_{0}$, and attackers, $\boldsymbol{a}_{-0}\triangleq\{\boldsymbol{z}^{(1)},\boldsymbol{z}^{(2)},...,\boldsymbol{z}^{(M)}\}$.

The Stackelberg solution concept is adequate for games with hierarchy in which the leader enforces its strategy and the followers respond, rationally (i.e. optimally), to the leader’s strategy [12]. We denote the optimal response of the attackers to action $\boldsymbol{a}_{0}$ played by the defender by $\mathcal{R}^{\textrm{att}}(\boldsymbol{a}_{0})\triangleq\{\boldsymbol{z}^{{(1)}^{*}}(\boldsymbol{a}_{0}),\boldsymbol{z}^{{(2)}^{*}}(\boldsymbol{a}_{0}),\dotsb,\boldsymbol{z}^{{(M)}^{*}}(\boldsymbol{a}_{0})\}$. This optimal strategy denotes the equilibrium strategy profile of the attackers as a response to the defender’s strategy. In this regard, $\boldsymbol{a}_{0}^{*}\in\mathcal{A}_{0}$ is a Stackelberg equilibrium if it minimizes the leader’s (i.e. defender’s) utility function $U_{0}$. In other words,

A Stackelberg equilibrium is guaranteed to exist and be unique if the optimal response of the followers is unique in response to every action of the leader. However, Theorem 1 proves the existence of at least one GNE for the attackers’ game. Hence, the followers can have multiple optimal responses to a leaders strategy. In this case, the leader can rank the GNEs corresponding to each strategy based on their impact on its utility and retain the one that leads to the worst utility (i.e. maximal utility given that the defender is a utility minimzer). The leader then selects the policy that minimizes this maximal utility. This is known as a hierarchical equilibrium (HE) [12]. In other words, $\boldsymbol{a}_{0}\in\mathcal{A}_{0}$ is a hierarchical equilibrium strategy for the defender if:

Here, we provide a methodology for finding a hierarchical equilibrium of the defender-attackers game as defined by (43).

We first consider the attackers subgame. To find an equilibrium that can be reached by the attackers, we propose a distributed learning algorithm, based on the framework of learning automata that was first analyzed in  [17]. The main drivers behind this algorithm are as follows. First, this algorithm is fully distributed in the sense that each attacker is only required to know its own action space, and not the shared one, and the observation of its own payoff after choosing an action. In this regard, knowledge of the action spaces of other attackers or even their existence is not required. Second, since the attackers’ game might admit multiple GNEs, the use of this algorithm, emulating practical smart grids security settings, enables the characterization of the GNE(s) that can be actually reached in practice.

The proposed learning algorithm is shown in Algorithm 1. In this algorithm, each attacker $m$ first initializes a strategy vector $\boldsymbol{q}^{(m)}$ containing a probability distribution over its attack space¹¹1This algorithm requires decritization of the action space of the attackers; our discretization approach is provided in Section V.. For instance, $q^{(m)}_{\boldsymbol{z}^{(m)}}(t)$ corresponds to the probability that attacker $m$ chooses attack $\boldsymbol{z}^{(m)}$ at time instant $t$. Then, at time instant $t$, each attacker chooses an attack randomly and independently from its attack space following the probability distribution available through its strategy vector. The collection of the attackers’ actions at time instant $t$ results in a payoff for each attacker denoted by $r_{m}(t)$. $r_{m}(t)$ is a positive normalized value which corresponds to a mapping from $[U_{m}^{\textrm{min}},U_{m}^{\textrm{max}}]\rightarrow[0,1]$ where $U_{m}^{\textrm{min}}$ and $U_{m}^{\textrm{max}}$ are the minimum and maximum achievable utilities by $m$. Based on the payoff that it receives at time instant $i$, each attacker, $m\in\mathcal{M}$, updates its strategy vector as follows:

where $b$ is an arbitrarily small positive constant and $\boldsymbol{e}^{(m)}(t)$ is a column vector of length equal to the size of the action set of attacker $m$. $\boldsymbol{e}^{(m)}(t)$ is a vector whose elements are equal to 0 except for the element corresponding to the action that was selected at time instant $t$. The element corresponding to the selected action will have a value of 1. Thus, given that the $j^{th}$ attack was selected by attacker $m$ at time instant $t$; then, $e^{(m)}_{j}(t)=1$ and $e^{(m)}_{k}(t)=0$ for $k\neq j$. This updating scheme is known as a linear reward-inaction ($L_{R-I}$) scheme [17]. Hence, with every iteration, the strategy vector of each attacker is updated and the algorithm repeats until each of the attackers’ strategy vectors has all elements equal to 0 except for one element which is equal to 1. Such a strategy vector shows which of the strategies is to be chosen by each attacker. The collection of these attacks (having a probability of 1 each) corresponds to the game’s equilibrium.

This algorithm has been discussed in[17, 18] where it has been proven that, for an arbitrarily small $b$, this algorithm asymptotically converges to a pure strategy Nash equilibrium (PSNE) when the game admits a PSNE.

Thus, the main difference between a GNE and a PSNE is that the GNE is an optimal action profile in which each action does not violate coupled constraints with other players. Thus, given that Algorithm 1 is guaranteed to converge to a PSNE, proving that this PSNE will never violate the coupled constraints is enough to prove the convergence to a GNE.

To be able to choose a hierarchical equilibrium strategy, a defender needs to anticipate the worst case GNE of the attacker to each defense strategy. This anticipation can be done through: i) repeating the learning algorithm of the attackers, Algorithm 1, starting from different initial conditions to find all possible GNEs from which worst case GNEs can be extracted, ii) using one of the various algorithms tailored to find solutions of GNE problems [15, 13, 16], iii) analytical derivation based on its full knowledge of the cyber-physical system model, energy market model and available past data.

When being able to anticipate all worst case GNEs of the attackers, the defender chooses the strategy that results in the best worst case GNE. This strategy and its corresponding GNE corresponds to the HE of the game.