Learning to Seek: Multi-Agent Online Source Seeking Against Non-Stochastic Disturbances

As mentioned earlier, the predominate approaches to solve the source seeking problem, including the well-known technique of extremum seeking control [10, 11], often build on the environment gradient estimation. These approaches can be indeed viewed as variants of the first-order optimization algorithm, which drive the agent to search for the local extremum values. In particular, by modeling the unknown environment as a time-invariant and concave real-valued function, the authors in [12] designed the distributed source seeking control law for a group of cooperative agents. Besides, the diffusion process is further considered in [13] for investigating the scenario of dynamical environments. The source seeking problems are also studied in [14, 15] by forcing the multiple agents to follow a specific circular formation. In addition, the stochastic gradient based methods are proposed in [16, 17] when considering that the gradient estimation is subject to environment and/or measurement noises. We should note that, also inherited from the first-order optimization algorithm, the above gradient based methods are very likely to be stuck at local extremum points when the considered environment is non-convex/non-concave. Furthermore, the gradient estimation is also sensitive to the measurement/environment noise, and thus additional statistical properties on the noise, such as known distribution with zero-mean, need to be imposed as assumptions in the problem setup.

Whilst it is unknown how to deal with noises without statistical properties in the context of source seeking, non-stochastic disturbance has been considered increasingly broadly in control communities. Within the classical robust control framework, the non-stochastic disturbance is often treated by considering the worst-scenario performance, see e.g., [18]. However, more recent works related to the learning based control mainly concern about the development of adaptive approaches which aim at controlling typically a linear system with adversarial disturbances while optimizing certain objective function with respect to the system states and control inputs[19, 20, 21, 22, 23]. To measure the performance of adaptive controllers, the notion of regret is adopted; that is, to measure the discrepancy between the gain/cost of the devised controller and that of the best one in hindsight. In particular, the authors in [19] devise the first $\mathcal{O}(\sqrt{K})$-regret algorithm by assuming a convex cost function and known system dynamics. Afterwards, such a regret bound is enhanced to be logarithmic in [20, 21] within the same problem setup. To further relax the requirement of the known dynamics, the authors in [22] develop the algorithm which attains $\mathcal{O}(K^{2/3})$-regret, and such a bound is also improved to $\mathcal{O}(\sqrt{K})$ later in [21, 23]. Though the above works have investigated quite thoroughly the non-stochastic setting in the context of learning based control, we remark that our paper considers a different problem where some standard conditions in control, such as controllability and observability, can be no longer simply assumed. In fact, our problem is more related to a sequential decision process; that is, the agents make their source seeking decisions in sequence while interplaying with perception of the unknown environment.

This sequential feature also makes our setting closely related to the well-known problem of multi-armed bandits. Therefore, another rich line of relevant works is on the series of bandit algorithms. More specifically, involved with the non-stochastic disturbances, linear bandits are investigated within two settings of non-stationary environment and adversarial corruptions, respectively. While the former one interprets the non-stochastic disturbance as a variation of the environment, the latter one is corresponding to corruptions injected by potential adversaries. Both cases are well studied in literature with the development of algorithms guaranteeing sub-linear regrets. To deal with the environmental non-stationarity, the WindowUCB algorithm is first proposed in [24] along with the technique of sliding-window least squares. It is shown that the algorithm achieves the regret of $\widetilde{\mathcal{O}}(K^{2/3}B_{K}^{1/3})$ where $B_{K}$ is a measure to the level of non-stationarity. The same regret is proved for the weighted linear bandit algorithm proposed in [25], which leverages on the weighted least-square estimator. Further, a simple restart strategy is developed in [26], obtaining the same regret. It is indeed proved that the $\widetilde{\mathcal{O}}(K^{2/3}B_{K}^{1/3})$-regret is the optimal one that can be achieved in the setting of non-stationary bandits. In terms of the adversarial bandits, a robust algorithm is proposed in [27] which guarantees the $\widetilde{\mathcal{O}}(B_{K}{K}^{3/4})$-regret, and thus it is sub-linear only if the level of adversarial corruptions satisfies $B_{K}=o(K^{1/4})$. More recently, such a regret has been improved to $\widetilde{\mathcal{O}}(B_{K}+\sqrt{K})$ in [28, 29] which is also shown to be nearly optimal in the adversarial setting. It can be concluded from the above discussion that, once $B_{K}$ grows sub-linearly, the regrets in both cases are guaranteed to be sub-linear. These are also the state-of-the-art that we are expected to achieve for our algorithm to be developed in this work.

This paper proposes an online source seeking algorithmic framework using the emerging learning technique, which is capable of i) dealing with the unknown environment in the presence of non-stochastic disturbances; and ii) taking advantages of the cooperation among the multi-agent network. In terms of the non-stochastic disturbances, two specific types of them are considered: i) an external one which disturbs the measurable states of the environment; and ii) an internal one which is truly evolved with the environment dynamics. To deal with them, an unified technique of discounted Kalman filtering is proposed to estimate the unknown environment states while mitigating the disturbances. Meanwhile, to build the cooperation among multiple agents and avoid the combinatorial complexity, we leverage the polytope confidence set, and as a result, the proposed algorithm is exceptionally computation-efficient in the multi-agent setting. It is shown by the regret analysis that our algorithm attains sub-linear regrets against both types of non-stochastic disturbances. The obtained two regrets are both comparable to the state-of-the-art in the studies of non-stationary and adversarial bandit algorithms. At last, all theoretical findings are validated by simulation examples on a real-world pollution monitoring application.

Consider an obstacle-free environment which is assumed to be bounded and discretized by a finite set of points $\mathcal{S}$ where each $\mathbf{s}\in\mathcal{S}$ represents the corresponding position. Suppose that the unknown state of the environment at each discrete time $k$ is described by a real-valued function $\phi_{k}(\cdot):\mathcal{S}\to\mathbb{R}_{+}$ which maps the positional information $\mathbf{s}$ to a positive quantity $\phi_{k}(\mathbf{s})$ indicating the environmental value of interest. Let us denote $N$ the total number of all points, i.e., $N=|\mathcal{S}|$, and for simplicity, denote $\bm{\phi}_{k}\in\mathbb{R}_{+}^{N}$ the vector which stacks all individual $\phi_{k}(\mathbf{s})$. Further, to characterize dynamics of the changing environment, we consider that the evolution of state $\bm{\phi}_{k}$ is basically governed by the following nominal linear time-varying (LTV) model

where the state transition matrix $A_{k}\in\mathbb{R}^{N\times N}$ is assumed to be known a prior. In order for the considered source seeking problem to be well-defined, we need the state $\bm{\phi}_{k}$ to be neither explosive nor vanishing to zero, which can be ensured by the following assumption.

We should note that the above Assumption 1 not only helps confine the behavior of the environment states, but also implies the invertibility of the state transition matrices $A_{k}$’s which aids the subsequent regret analysis of our algorithm. In fact, such an assumption is not unusual in the study of system control and estimation problems; see e.g., [30, 31, 32, 33].

Now, in order to further impose the underlying disturbances into the environment model, let us consider the following two types of non-stochastic ones on top of the nominal dynamics:

Note that in both types, $\widetilde{\bm{\phi}}_{k}\in\mathbb{R}_{+}^{N}$ denotes the disturbed state. However, while the first type of disturbance can be interpreted as an external one since $\bm{\phi}_{k}$ in (3a) is still evolved according to the nominal dynamics (1) and the disturbance $\bm{\delta}_{k}$ only affects the state $\widetilde{\bm{\phi}}_{k+1}$ in one step, the second type can be viewed as an internal one since the disturbance $\bm{\delta}_{k}$ is intrinsically imposed into the dynamics and accumulated during the evolution of $\widetilde{\bm{\phi}}_{k}$. In fact, we shall remark that the two types of disturbances both find a wide range of real-world applications. For instance, in the scenario of pollution monitoring which is investigated in our simulations, the external disturbance could correspond to certain unrelated emitters which will not change the locations of sources of interest but interfere the perceptible environment states, the internal one might result from some environmental conditions, such as wind, which will truly affect the diffusion of pollutants and thus change their positions. It is also enlightened by the provided example that the localization of sources should be considered differently for the above two cases. More details will be found in Section II-B. In addition, we note that the internal disturbance can be also used to capture to some extent the unmodeled dynamics of the unknown environment. However, no matter which type of disturbances is involved in the process, only the disturbed state $\widetilde{\bm{\phi}}_{k}$ is measurable for the agents which are employed to operate in the environment later.

As we have remarked earlier, the disturbances of both types are supposed to be non-stochastic, i.e., no statistical property in any form is assumed regarding $\bm{\delta}_{k}$. Instead, to characterize the effect of both disturbances in long term, we consider to impose the following assumption.

With the aim to locate the potential sources which usually correspond to the extreme values in the unknown environment state, we deploy a network of $I$ agents and expect each of them $i\in\mathcal{I}:=\{1,2,\cdots,I\}$ to seek its best positions $\mathbf{p}_{k}^{\star}[i]\in\mathcal{S}$ at each time $k$ by solving the following maximization problem,

Notice that the summation involved in the objective function $F_{k}(\cdot):\mathcal{S}^{I}\to\mathbb{R}_{+}$ takes into account the union of positions $\mathbf{p}[i]$’s, therefore all agents will naturally tend to locate as many distinct positions as possible for the purpose of maximizing $F_{k}(\cdot)$. In addition, it is now clear to see the reason why Assumption 1 would be needed, i.e., the maximization in (5) is otherwise not well-defined if the environment state $\bm{\phi}_{k}$ explodes or vanishes to zero. Further, we should also note that an inherent difference will take place in the counted state $\bm{\phi}_{k}$ involved in the objective function when considering the disturbances of the two types. More precisely, for the first type of disturbance, i.e., the external one, the positions of sources should be indeed reflected by the undisturbed $\bm{\phi}_{k}$, though only the information of disturbed $\widetilde{\bm{\phi}}_{k}$ is measurable for agents. On the contrary, for the second type, i.e., the internal disturbance, the disturbed $\widetilde{\bm{\phi}}_{k}$ should be taken into account in (5), since $\bm{\delta}_{k}$ is evolved in the environment dynamics and changes the positions of sources. On this account, we emphasize that while the maximization problem (5) is precisely the one that the agents would like to solve when considering the external disturbance, yet for the internal one, the objective function should be amended as

With the above difference presented in the objective functions, the main challenges in solving the maximization problems are also distinguishable in principle. Whilst the former is to extract the true information hidden in $\bm{\phi}_{k}$ in the case that only $\widetilde{\bm{\phi}}_{k}$ is accessible, the latter is to identify and compensate the unmodeled disturbance $\bm{\delta}_{k}$. Despite this difference, we develop in this paper an unified algorithmic framework for both cases, enabling the agents to track the dynamical sources in an online manner. We remark that this is also one of the main contributions of our work.

Another common technical issue, regardless of types of the disturbances involved, is to deal with the estimation of the environment. Therefore, we leverage on the following linear stochastic measurement model,

where $\mathbf{z}_{k}^{i}\in\mathbb{R}^{m}$ is the $i$-th agent’s obtained measurement at the time-step $k$; $H^{i}\big{(}\mathbf{p}_{k}[i]\big{)}\in\mathbb{R}^{m\times N}$ denotes the measurement matrix depending on the agent’s position $\mathbf{p}_{k}[i]$; and $\mathbf{n}_{k}^{i}\in\mathbb{R}^{m}$ is the measurement noise which is assumed to be independent and identically distributed (i.i.d.) Gaussian with zero mean and variance $V^{i}=v^{i}\cdot\mathbf{I}_{m}$. We shall note that the measurement matrix $H^{i}\big{(}\mathbf{p}_{k}[i]\big{)}$ is not specified in (7). In fact, it can be defined by various means based on the agent’s position. Nevertheless, we assume that each $H^{i}\big{(}\mathbf{p}_{k}[i]\big{)}$ has the following basic form,

where $\mathbf{e}_{l}$ denotes the unit vector, i.e., the $l$-th column of the identity matrix, and $\mathcal{C}_{k}^{i}$ is the set of positions which are covered by the agent’s sensing area at the time-step $k$. It is natural to assume that the position where the agent currently locates falls into its sensing area, i.e., $\mathbf{p}_{k}[i]\in\mathcal{C}_{k}^{i}$.

According to the measurement model (7) introduced in the previous section, let us first express it as a compact form which counts for all agents within the network. For this purpose, we stack all the measurements $\mathbf{z}_{k}^{i}$’s and also the noise $\mathbf{n}_{k}^{i}$’s as the concatenated vectors $\mathbf{z}_{k}\in\mathbb{R}^{M}$ and $\mathbf{n}_{k}\in\mathbb{R}^{M}$ with $M=mI$, e.g., $\mathbf{z}_{k}:=[(\mathbf{z}^{1}_{k})^{\top},(\mathbf{z}^{2}_{k})^{\top},\cdots,(\mathbf{z}^{I}_{k})^{\top}]^{\top}\in\mathbb{R}^{M}$. Likewise, we define the concatenated measurement matrix $H_{k}\in\mathbb{R}^{M\times N}$ by stacking all local $H^{i}(\mathbf{p}_{k}[i])$’s. Consequently, the measurement model of the compact form can be written as

Note that, in the notation $H_{k}$, we have absorbed for simplicity the dependency on agents’ positions $\mathbf{p}_{k}[i]$’s into the index $k$. In addition, by our assumption on the measurement noise, one can have that the concatenated noise $\mathbf{n}_{k}$ is also i.i.d. Gaussian with zero-mean and variance being

Equipped with the agents’ measurement model in its compact form (9), we are now ready to present the technique of discounted Kalman filtering. Similar to the standard Kalman filter, we also use mean $\widehat{\bm{\phi}}_{k}\in\mathbb{R}^{N}$ and covariance $\Sigma_{k}\in\mathbb{R}^{N\times N}$ to recursively generate estimates of the unknown environment. However, a primary difference is that two positive sequences of weights $\{\lambda_{k}\}_{k\in\mathbb{N}_{+}}$ and $\{\omega_{k}\}_{k\in\mathbb{N}_{+}}$ are imposed in the filtering process with the aim to mitigate the effect of disturbances presented in the environment. Keep this in mind, the discounted Kalman filter performs the following recursions,

Notice that $\Sigma_{k+\scriptscriptstyle{1/2}}\in\mathbb{R}^{N\times N}$ and $\widehat{\bm{\phi}}_{k+\scriptscriptstyle{1/2}}\in\mathbb{R}^{N}$ here denote the intermediate results during the recursions; $\Gamma_{k}\in\mathbb{R}^{N\times N}$ is an auxiliary matrix initialized by $\Gamma_{0}=\mathbf{I}_{N}$; and the variables $\mathbf{y}_{k}:=H_{k}^{\top}V^{-1}\mathbf{z}_{k}\in\mathbb{R}^{N}$ and $Y_{k}:=H_{k}^{\top}V^{-1}H_{k}\in\mathbb{R}^{N\times N}$, which can be readily acquired by consensus schemes, i.e., [35], incorporate latest measurements into the update of estimates. Next, to better show how the imposed weights help deal with the non-stochastic disturbances, we present in the subsequent lemma another expression of the discounted Kalman filter (11).

Based on $\widehat{\bm{\phi}}_{k}$ and $\Sigma_{k}$, we now introduce the key notion of D-UCB $\bm{\mu}_{k}\in\mathbb{R}^{N}$, which is defined as follows,

Note that the operator $\text{diag}^{1/2}(\cdot):\mathbb{R}^{N\times N}\to\mathbb{R}^{N}$ maps the square root of the matrix diagonal elements to a vector, and the sequence $\{\beta_{k}(\delta)\}_{k\in\mathbb{N}_{+}}$ depending on a predefined confidence level $\delta$ will be specified subsequently in the next section. With the aid of the defined D-UCB $\bm{\mu}_{k}$, one can update the agents’ seeking positions in an online manner, by solving the following maximization problem:

Here, $\mathbf{p}_{k}\in\mathcal{S}^{I}$ stacks the decided seeking positions $\mathbf{p}[i]_{k}$’s for all agents, and likewise, ${\mu}_{k}(\mathbf{s})\in\mathbb{R}$ represents one component of the vector ${\bm{\mu}}_{k}$ which corresponds to the position $\mathbf{s}\in\mathcal{S}$. The complete multi-agent online source seeking scheme under the environment with disturbances is outlined in Algorithm 1.

As we have informed in the previous discussion, for the first type of disturbance, the objective function $F_{k}(\cdot)$ in (5) takes into account the undisturbed state $\bm{\phi}_{k}$. Therefore, we introduce the notion of regret for the first case as follows,

where $\mathbf{p}_{k}^{\star}$ denotes the optimal solution to problem (5) and $\mathbf{p}_{k}$ corresponds to the decision generated by our source seeking algorithm. Here, we aim to show that the cumulative regret, i.e., $R_{K}:=\sum_{k=0}^{K}r_{k}$, increases sub-linearly with respect to the number of time-steps $K$, namely the regret $r_{k}$ converges to zero on average. To this end, let us first show the following result which formalizes that the D-UCB $\bm{\mu}_{k}$ provides indeed a valid upper bound for the unknown state $\bm{\phi}_{k}$.

It can be concluded from Proposition 1 that the D-UCB $\bm{\mu}_{k}$ is guaranteed to be an upper bound for $\bm{\phi}_{k}$ with probability at least $1-\delta$. In fact, considering that the disturbance of type I is not really evolved in the environment dynamics, the weight $\omega_{k}$ is thus set to be zero during the whole process. Further, to extract the true information, we set the weight $\lambda_{k}$ adaptively according to the timely estimation of the environment. Since the estimate covariance $\Sigma_{k}$ is, in general, decreased as more measurements are absorbed during the filtering process, it can be seen that the sequence $\{\lambda_{k}\}_{k\in\mathbb{N}_{+}}$ will increase with an upper bound set to be $\bar{\lambda}$. With the help of Proposition 1, we are now ready to present the result of regret analysis for our algorithm.

Similar to the previous analysis, to provide the performance guarantee for our algorithm in this part, we also rely on the notion of regret. However, considering the difference in features of the disturbance of type II; see details in Section II-B, the definition of regret should be amended accordingly,

where the objective function $\widetilde{F}_{k}(\cdot)$ is defined in (6). Likewise, it is also expected to show a sub-linear cumulative regret, i.e., $\widetilde{R}_{K}:=\sum_{k=0}^{K}\widetilde{r}_{k}$ increases sub-linearly with respect to $K$.

Due to the fact that the disturbance of type II is imposed in the environment dynamics, the current state $\widetilde{\bm{\phi}}_{k}$ inherently accumulates all disturbances prior to the time $k$. As a result, it is not necessarily implied by Assumptions 1 and 2 that $\widetilde{\bm{\phi}}_{k}$ is upper bounded if the sequence $B_{K}$ is allowed to be increased infinitely. Thus, to ensure the well-definedness of our problem, we need an additional assumption.

Now, we follow the similar path as in the previous analysis to show the sub-linear regret of $\widetilde{R}_{K}$. Note that, due to the long term effect of the second type disturbance in the state $\widetilde{\bm{\phi}}_{k}$, one cannot expect that the D-UCB $\bm{\mu}_{k}$ serves as an upper bound for $\widetilde{\bm{\phi}}_{k}$. To deal with this issue, we construct an auxiliary variable $\mkern 1.5mu\overline{\mkern-1.5mu\bm{\phi}\mkern-1.5mu}\mkern 1.5mu_{k}\in\mathbb{R}^{N}$, i.e.,

which helps build a connection between $\bm{\mu}_{k}$ and the state $\widetilde{\bm{\phi}}_{k}$ as shown in the following propositions.

It is proved by Proposition 2 that the D-UCB $\bm{\mu}_{k}$ provides a valid upper bound for the constructed variable $\mkern 1.5mu\overline{\mkern-1.5mu\bm{\phi}\mkern-1.5mu}\mkern 1.5mu_{k}$ if $\beta_{k}(\delta)$ is chosen appropriately. To further build the connection between $\bm{\mu}_{k}$ and the true state $\widetilde{\bm{\phi}}_{k}$, it can be shown that the discrepancy between $\mkern 1.5mu\overline{\mkern-1.5mu\bm{\phi}\mkern-1.5mu}\mkern 1.5mu_{k}$ and $\widetilde{\bm{\phi}}_{k}$ will be bounded by a term related to the disturbances $\bm{\delta}_{k}$. However, for the sake of presentation, such a result will be deferred, and we directly provide the statement of sub-linear regret for our algorithm in the following theorem. The bound of $\|\mkern 1.5mu\overline{\mkern-1.5mu\bm{\phi}\mkern-1.5mu}\mkern 1.5mu_{k}-\widetilde{\bm{\phi}}_{k}\|$ will be shown as an intermediate step of proofs for the theorem in Appendix.

Before the end of this section, a few more remarks should be added on the obtained results of the above regret analysis.

First, to tackle with the two types of non-stochastic disturbances, it can be seen from the propositions that the sequences of weights $\lambda_{k}$ and $\omega_{k}$ are also determined differently. More specifically, for the external disturbance which only affects the measured state $\widetilde{\bm{\phi}}_{k}$ but not evolve with the nominal dynamics, the sequence of $\lambda_{k}$ is chosen as increased at the same rate of $1/\|Y_{k}\|_{\Sigma_{k}}$. This is due to the fact that the disturbance $\bm{\delta}_{k}$ in this case only comes into play when the state is measured, and thus the weight $\lambda_{k}$ is also adjusted according to the measurement information in $Y_{k}$ and the current progress on $\Sigma_{k}$. Since the covariance $\Sigma_{k}$, which basically suggests the uncertainty of our estimation, decreases as more measurements are absorbed in the estimation, the weight $\lambda_{k}$ is increased during the process, meaning that the measurement received later are more trusted. For the internal disturbance, the sequence of $\lambda_{k}$ is also chosen as increased, but at a fixed exponential rate of $(1/\gamma)^{k}$. Another primary difference is that, while $\omega_{k}$ is set to be zero previously, here we let $\omega_{k}$ increase at the same exponential rate of $(1/\gamma)^{k}$. The reason for such a difference can be explained as follows. Since the internal disturbance, regardless of the measurements, is accumulated during the whole process, an additional weight needs to be incorporated to deal with it globally, and therefore the increasing $\omega_{k}$ is introduced to decrease the covariance $\Sigma_{k}$ accordingly. Note that this does not mean the uncertainty of our estimation is decreased brutally, as in the D-UCB $\bm{\mu}_{k}$, the sequence of $\beta_{k}(\delta)$ is also increased by an extra term related to $1/\gamma$ to adjust our construction of the confidence bound.

Second, it can be concluded by the two theorems that once the disturbance bound $B_{K}$ increases sub-linearly, the regrets generated by our algorithm for both cases also grow at a sub-linear rate, meaning that the agents will be able to track the moving sources dynamically under the disturbed environment. More precisely, while the regret for the first case increases at the rate of $\widetilde{\mathcal{O}}(\sqrt{K}+B_{K})$, the rate is $\widetilde{\mathcal{O}}(B_{K}^{1/3}K^{2/3})$ for the second case. Note that both of them are identical to the state-of-the-art results in the study of bandit algorithms with non-stationary and adversarial settings. Therefore, we can conclude that our developments of discounted Kalman filter and D-UCB do not degrade the performance of algorithm with respect to its convergence. However, in terms of the scale of the problem $N$, i.e., size of the searching environment, the complexity of our algorithm indeed grows at the rate of $\widetilde{\mathcal{O}}(N^{2})$, as compared to $\widetilde{\mathcal{O}}(N)$ in the literature. This is mainly due to the reason that the ellipsoid confidence sets in the classical UCB-based methods is changed to the polytope one in our algorithm. Despite this fact, we argue that such an increase of complexity is actually reasonable, since more computational complexities have been reduced by avoiding the combinatorial problems at each step.

To prove the inequality (17) in Proposition 1, with the help of the above defined vector norms, it will suffice to show

Note that the inequality in (33) is stronger than the one in (17), in the sense that the state $\bm{\phi}_{k}$ is both upper and lower bounded. Though the lower bound is not reflected in the development of our algorithm, it helps the proof for the sub-linear regret. In addition, due to the fact that the weight $\omega$ is specified as $\omega_{k}\equiv 0$ in this part, it indeed changes the generation of state estimates by simplifying the matrix $\Upsilon_{k}$ as

According to the nature of the first type of disturbance, the disturbed state can be expressed as $\widetilde{\bm{\phi}}_{k}=A[k:1]\bm{\phi}_{0}+\bm{\delta}_{k}$ and thus the measurement is $\mathbf{z}_{k}=H_{k}(A[k:1]\bm{\phi}_{0}+\bm{\delta}_{k})+\mathbf{n}_{k}$. Then, by Lemma 1 and the definitions of $\Upsilon_{k}$ and $Y_{k}$, the state estimate $\widehat{\bm{\phi}}_{k}$ has

Therefore, it holds for $\forall\mathbf{x}\in\mathbb{R}^{N}$ that