Attack Impact Evaluation for Stochastic Control Systems through Alarm Flag State Augmentation

The attack impact evaluation problem for control systems has been considerably studied [8, 13, 12, 14, 15, 9, 16, 10, 17, 18, 19, 20, 21, 22]. These works formulate the problem as a constrained optimal control problem, but the computation approaches differ based on the type of system, detector, and the objective function used to quantify the attack impact. To the best of our knowledge, this study is the first work to handle general systems and detectors.

Our idea of alarm flag state augmentation is to add information sufficient for determining the optimal decision at each time step. A similar concept has been proposed in previous studies especially in the context of risk-averse Markov decision process (MDP) [23, 24, 25]. The work [23] treats a non-standard MDP where the objective function is given not by expectation but by conditional-value-at-risk (CVaR), to which value iteration can be applied by considering an augmented state space for CVaR. In [24], this idea is generalized to chance-constrained MDP. The work [25] proposes risk-aware reinforcement learning based on state space augmentation. Moreover, linear temporal logic specification techniques can handle general properties, such as safety, invariant, and liveness for discrete event systems [26]. However, our study provides a clear interpretation of the augmented state in the context of control system security, leading to a reasonable extension to the multi-alarm avoidance problem discussed in Sec. IV. Additionally, we consider a continuous state space, whereas existing studies mainly focus on finite or discrete state spaces.

Temporally joint chance constraints in optimal control have also been studied [27, 28, 29], but these methods rely on approximating the chance constraint. Furthermore, they do not discuss the state space augmentation of the decision at each time step. Finally, a continuous-time optimal control problem with a joint-chance constraint is considered in [30, 31] although the process stops once the state reaches the unsafe region in their formulation.

A preliminary version of this note has been presented in [32], but it only considers finite MDPs, and the current study extends it to continuous spaces. Furthermore, the previous study did not included the theoretical claims and the proofs of Propositions 1 and 2 in this note.

This note is organized as follows. Sec. II defines the system model, clarifies the threat model, and formulates the attack impact evaluation problem. In Sec. III, the difficulty of the formulated problem is explained. Subsequently, we propose a problem reformulation in a tractable form by introducing the alarm flag state augmentation. Sec. IV first provides a characterization of the optimal policy after an alarm is triggered. Based on the observation, we extend the formulation to be able to handle multi-alarm avoidance policies and show that the proposed idea is still valid in the extended variant. In Sec. V, the theoretical results are verified through numerical simulation, and finally, Sec. VI concludes and summarizes this note.

We denote the set of real numbers by $\mathbb{R}$, the $n$-dimensional Euclidean space by $\mathbb{R}^{n}$, the $t$-ary Cartesian power of the set $\mathcal{X}$ for a positive integer $t$ by $\mathcal{X}^{t}$, the complement of a set $\mathcal{X}$ by $\mathcal{X}^{\rm c}$, the tuple $(x_{0},\ldots,x_{t})$ by $x_{0:t}$, and the Borel algebra of a topological space $\mathcal{X}$ by $\mathcal{B}_{\mathcal{X}}$.

Consider a discrete-time nonlinear stochastic control system of the form

with $f:\mathcal{X}\times\mathcal{A}\times\mathcal{W}\to\mathcal{X}$ where $x_{t}\in\mathcal{X}\subset\mathbb{R}^{n}$ is the state, $a_{t}\in\mathcal{A}\subset\mathbb{R}^{m}$ is the attack signal, and $w_{t}\in\mathcal{W}\subset\mathbb{R}^{\ell}$ is an independent random process noise. The distribution of the initial state is denoted by $p_{0}(dx_{0})$. An attack detector equipped with the control system triggers an alarm when the state reaches the alarm region $\mathcal{X}_{\rm a}\subset\mathcal{X}$.

Remark: This model includes control systems with the typical cascade structure illustrated by Fig. 1. Let the dynamics of the control system $\Sigma$ and the detector $\mathcal{D}$ be given by

respectively, with the state spaces $\mathcal{Z}$ and $\mathcal{Z}^{\rm d}$. The binary signal $\delta_{t}\in\{0,1\}$ describes whether an alarm is triggered, or not, at the time step. It is clear that the cascade system can be described in the form by taking $\mathcal{X}:=\mathcal{Z}\times\mathcal{Z}^{\rm d}$ and $\mathcal{X}_{\rm a}:=h^{-1}(\{1\})$.

In this study, we consider the following threat model:

• •

  The adversary has succeeded in intruding into the system and can execute any attack signal in a probabilistic manner at every time step.

• •

  The adversary has perfect model knowledge.

• •

  The adversary possesses infinite memory and computation resources.

• •

  The adversary can observe the state at every time step.

• •

  The attack begins at $t=0$ and ends at $t=T-1$.

The threat model implies that the adversary can implement an arbitrary history-dependent randomized policy $\pi\in\Pi^{\rm h}$, where $\pi=(\pi_{t})_{t=0}^{T-1}$ is a tuple of policies at every time step. Let $h_{t}\in\mathcal{H}_{t}:=(\mathcal{X}\times\mathcal{A})^{t-1}\times\mathcal{X}$ denote the history. The policy at each time step is a stochastic kernel on $\mathcal{A}$ given $\mathcal{H}_{t}$ denoted by $\pi_{t}(da_{t}|x_{0:t},a_{0:t-1})$. It is well known that a policy $\pi$ uniquely induces a probability measure $\mathbb{P}^{\pi}$ on $\mathcal{H}_{T}$ [33, Chap. 11]. The probabilities are specifically given by

where $X_{t}\in\mathcal{B}_{\mathcal{X}}$ and $A_{t}\in\mathcal{B}_{\mathcal{A}}$ and $p_{t}(dx_{t}|x_{t-1},a_{t-1})$ is the state transition kernel on $\mathcal{X}$ given $\mathcal{X}\times\mathcal{A}$ induced by $f$ and the distribution of $w_{t-1}$ [34, Chap. 8]. The expectation operator with respect to $\mathbb{P}^{\pi}$ is denoted by $\mathbb{E}^{\pi}$.

The objective of the adversary is to maximize a cumulative attack impact while avoiding detection. Let the impact be quantified as

The adversary keeps herself stealthy over the considered time period $\mathcal{T}:=\{0,\ldots,T\}$. Specifically, the probability the event that an alarm is triggered at some time step defined by

is made less than or equal to a given constant $\Delta\geq 0$.

The attack impact evaluation problem is formulated as a stochastic optimal control problem with a temporally joint chance constraint:

In the subsequent section, we explain its difficulty and propose an equivalent reformulation in a tractable form.

It is well known that Markov policies, which depend only on the current state, can attain the optimal value for unconstrained stochastic optimal control problems [34, Proposition 8.1]. However, Problem 1 has a temporally joint chance constraint that cannot be decomposed with respect to time steps. Hence Markov policies cannot attain the optimal value in general, an example of which is provided in the Appendix. As a result, the size of the search space grows exponentially with the time horizon length, making the problem intractable even for small instances.

The key idea in this paper to overcome this challenge is to augment the alarm history information into the state space. We define the augmented state space and the induced augmented system next.

The augmented state $f_{t}\in\mathcal{F}$ is referred to as the alarm flag, since $f_{t}=1$ indicates that the alarm has been triggered before the time step $t\in\mathcal{T}$, whereas $f_{t}=0$ indicates otherwise. For the augmented system, we denote the set of histories by $\hat{\mathcal{H}}_{t}:=(\mathcal{X}\times\mathcal{F}\times\mathcal{A})^{t-1}\times(\mathcal{X}\times\mathcal{F})$, the set of history-dependent randomized policies by $\hat{\Pi}^{\rm h}$, the probability measure on $\hat{\mathcal{H}}_{T}$ induced by $\hat{\pi}\in\hat{\Pi}^{\rm h}$ by $\mathbb{P}^{\hat{\pi}}$, and the expectation operator with respect to $\mathbb{P}^{\hat{\pi}}$ by $\mathbb{E}^{\hat{\pi}}$.

By using the alarm flag, we can rewrite the temporally joint chance constraint in (1) as an isolated chance constraint on the state only at the final time step. It is intuitively true that $(\lor_{t\in\mathcal{T}}X_{t}\in\mathcal{X}_{\rm a})$, the event that an alarm is triggered at some time step, is equivalent to $f_{T}^{-1}(\{1\})$, the event that the alarm flag takes the value $1$ at the final time step. This idea yields the reformulated problem

The most significant aspect of this formulation is that the chance constraint depends on the marginal distribution with respect to the final time step only. Hence, the optimal value of (2) can be attained by Markov policies for augmented state space $\hat{\mathcal{X}}$ [34, Proposition 8.1]. Thus, the problem (2) can be reduced to

where the search space is replaced with $\hat{\Pi}^{\rm m}$, the set of Markov policies for the augmented system.

We justify the reformulation in a formal manner. First, we show the following lemma.

Lemma 1 implies that the stochastic behaviors of the original system and the augmented one are identical with appropriate policies related to each other through (4).

The following theorem is the main result of this paper.

Theorem 1 justifies the reformulation from (1) to (3). The reformulated problem (3) can be solved using existing numerical methods if the dimension of the spaces is not too large [35].

We observe that the optimal policy after an alarm is triggered is characterized using an optimal policy of an unconstrained problem. Consider the problem

and assume that there exists a unique optimal Markov policy, denoted by $\pi^{{\rm u}\ast}$, for simplicity.

We first show the following lemma, which claims that the probability of the alarm flag is invariant as long as the policy conditioned by $f_{t}=0$ is invariant.

Based on Lemma 2, we can show the following proposition, which partially characterizes the optimal policy for (3).

for $t=0,\ldots,T-1$.

Proposition 1 implies that the adversary cares about being detected when there have been no alarms so far, but does no longer care once an alarm has been triggered. In reality, however, a single alarm may not result in counteractions by the defender due to the presence of false alarms, and a different strategy that avoids serial alarms can possibly be more reasonable. Therefore, it is more preferable to extend our problem formulation (1) to be able to handle multiple alarms.

We define the event that alarms are triggered more than or equal to $i$ times,

where

The extended version of the attack impact evaluation problem for multi-alarm avoidance strategies is formulated as follows.

The same idea of the alarm flag state augmentation proposed in Sec. III can be applied to Problem 2 as well by augmenting information on the number of alarms instead of the binary information. The augmented state space and the augmented system for Problem 2 are defined as follows.

The alarm number augmentation naturally leads to an equivalent problem

where the search space is the set of Markov policies for the state space augmented with the number of alarms. The following theorem is the correspondence of Theorem 1.

Moreover, the correspondence of Proposition 1 is described as follows.

Proposition 2 means that the adversary does not avoid detection after the number of alarms reaches $M$.

Remark: The constraint in the extended problem (7) restricts the probability distribution of the number of alarms. In other words, the formulation utilizes a risk measure on a probability distribution instead of a typical statistic. Several risk measures have been proposed, such as CVaR, which is one of the most commonly used coherent risk measures [36]. Those risk measures compress risk of a random variable with a distribution into a scalar value. Because our formulation uses the full information of the distribution, the constraint can be regarded as a fine-grained version of standard risk measures.

Consider the one-dimensional discrete-time integrator

with the CUSUM attack detector [37, Chap. 2]

with the bias $b^{\rm d}>0$ and the threshold $\tau^{\rm d}>0$, where $\mathcal{A}=\mathcal{W}=\mathbb{R}$. The state space $\mathcal{X}=\mathbb{R}^{2}$ and the alarm region $\mathcal{X}_{\rm a}$ are constructed according to Sec. II-A. The process noise follows the white Gaussian distribution with mean zero and variance $\sigma^{2}$. The adversary’s objective is to drive the system state around a reference value $z_{\rm ref}\in\mathbb{R}$. Accordingly, the objective function is set to a quadratic function

The constants are specifically set to $T=15,$ $\sigma=0.1,$ $b^{\rm d}=0.8,$ $\tau^{\rm d}=2,$ and $z_{\rm ref}=1.5$. We compute $\hat{\pi}^{\ast}_{t}(da_{t}|x_{t},0)$ based on discretization of the state and input spaces and use a standard linear programming approach for solving the resulting constrained finite MDP [38]. On the other hand, we analytically compute $\hat{\pi}^{\ast}_{t}(da_{t}|x_{t},1)$ as the unconstrained discrete-time linear quadratic regulator [39, Chap. 4] based on Proposition 1.

We first treat the formulation of Problem 1. Set the constant on the stealth condition by $\Delta=0.5$. Fig. 2 shows the simulation results with the optimal policy obtained by solving the equivalent problem (3). Figs. 2a and 2b depict the empirical means of $z_{t}$ and $z^{\rm d}_{t}$ conditioned by whether an alarm is triggered at least once during the process, or not, respectively. It can be observed that $z^{\rm d}_{t}$ increases even after an alarm is triggered, as claimed in Sec. IV-A. Fig. 2c depicts the probabilities with respect to the total number of alarms during the process. It can be observed that a large number of alarms occur with a high probability. The result indicates that the formulation of Problem 1 leads to a policy such that the number of alarms becomes large once an alarm is triggered.

We next treat the formulation of Problem 2. Set the constants on the stealth condition by

with $M=3$. Fig. 3 shows the simulation results. The subfigures correspond to those in Fig. 2. It can be observed that the trajectory of $z^{\rm d}_{t}$ is kept less than the detection threshold $\tau^{\rm d}$ over almost the entire period. Accordingly, the probability depicted in Fig. 3 suggests that the obtained policy avoids multiple alarms.