maximum principle for optimal control of interacting particle system: stochastic flow model

In this paper, we investigate a stochastic optimal control problem for a family of particles whose evolution is described by an equation with interaction. Equations with interactions were introduced by A.A. Dorogovtsev in [1, 2]. Here we use the following form of the stochastic differential equation (SDE) with interactions:

with $m-$dimensional standard Wiener process $\{W_{t}\}_{t\geq 0}$ and the function $\sigma$ with values in $\mathbb{R}^{d\times m}$. Equation (1) describes the dynamics of a family of particles $\{x(u,\cdot)\}_{u\in\mathbb{R}^{d}}$ that start from each point of the space $\mathbb{R}^{d}$. The initial mass distribution of the family of particles is denoted by $\mu_{0},$ which is supposed to be a probability measure on the Borel $\sigma-$field in $\mathbb{R}^{d}$. The mass distribution of the family of particles evolves over time, and at moment $t$, it can be written as the push forward of $\mu_{0}$ by the mapping $x(\cdot,t)$. The coefficients of drift and diffusion in Equation (1) depend on the mass distribution $\mu_{t}$. This gives the possibility to describe the motion of a particle that depends on the positions of all other particles in the system.

Analogously to the usual stochastic optimal control problem, one can state the following control problem for a SDE with interaction:

The goal of the stochastic optimal control problem is to find a control process
$\left(\alpha_{t}\right)_{0\leq t\leq T}$ that minimizes the cost function $J(\alpha)$ of the form

The time evolution of the system (2) depends upon a stochastic process $\mathbf{\alpha}=\left(\alpha_{t},\quad 0\leq t\leq T\right)$, which is supposed to be progressively measurable with respect to filtration $\{\mathcal{F}_{t}\}_{t\geq 0}$ of Wiener process. The value $\alpha_{t}$ should be considered as a control, the choice of which is made at time $t$. The investigation of such an optimal control problem was inspired by the Monge-Kantorovich [3, 4, 5], transport problem. The Monge-Kantorovich transport problem deals with a mapping of one probability measure to another, minimizing a transportation cost function. In the stochastic optimal control problem for equation with interaction, the system starts from the mass distribution $\mu_{0}$, and the cost function (3) of the control problem has two parts: a running cost function (integral of $f$ over time) and a terminal cost function $g$, that depends on mass distribution of the system at the final time. In terms of the Monge-Kantorovich transport problem, the function $g$ can play the role of a distance between a given measure $\nu$ and the distribution of mass in the system at the final time. For example, for a given probability measure $\nu$ we can take $g(X(u,T),\mu_{T})=\rho(\mu_{T},\nu),$ where the distance $\rho$ between measures can be given by

with non-negative continuous function $\Gamma$ such that $\Gamma(0)=0$.

In the case of a discrete initial measure $\mu_{0}=\sum_{i=1}^{n}p_{i}\delta_{u_{i}}$, $u_{i}\in\mathbb{R},1\leq i\leq n$, the mass distribution of the system is equal to $\mu_{t}=\sum_{i=1}^{n}p_{i}\delta_{x(u_{i},t)}.$ From this one can see that the dependence of coefficients on the measure $\mu_{t}$ in equation (2) and in the cost function (3) is realized through the dependence on the positions of the heavy points $x(u_{1},t),\ldots,x(u_{n},t).$ In this case the optimal control process $(\alpha_{t})_{0\leq t\leq T}$ for the problem (2)-(3) can be found from the corresponding $n-$dimensional stochastic optimal control problem for the system of heavy particles $x(u_{1},t),\ldots,x(u_{n},t)$ (see [6]). Having optimal control process $(\alpha_{t})_{0\leq t\leq T}$ we solve equation (2) for all points with zero mass $x(u,\cdot),\ u\in\mathbb{R}^{2}\setminus\{u_{1},\ldots,u_{n}\}.$ This example shows how we can describe the dynamic of a family of particles $\{x(u,t),\ u\in\mathbb{R}^{d},\ t\geq 0\}$ driven not only by external forces but also by internal forces. Precisely, the internal force is represented through the dependence of the coefficients in (2) on the position of heavy particles in the family of particles.

It should be mentioned that the considered SDEs with interaction and BSDEs with interaction is different from the Mckean-Vlasov equation, which is actively investigated by many researchers [7, 8, 9, 10]. First of all, the coefficients of the Mckean-Vlasov equation depend on the distribution law of the position of a particle, but not on the distribution of mass of a system, as in SDEs with interaction. More specifically, Mckean-Vlasov equation and SDEs with interaction describe the particle system in different views. For Mckean-Vlasov equation, there are a lot of particles driven by independent noises, and their initial positions are i.i.d variables. The distribution of the position for these particles converge to the probability distribution law. For SDEs with interaction, the initial positions of these particles are determined and the distribution of their initial position is known. There are not many independent noise. Such as these particles are driven by a Brownian sheet, which makes they have different random perturbations when they are in different locations. Consider the following example, we assume there are $N$ particles. For Mckean-Vlasov type, the state process has the form:

for $1\leq i\leq N$, where $x^{i}_{0}$ are i.i.d variables with the probability distribution $\mathcal{L}_{0}$ and $W_{t}^{i}$ are independent Brownian motions. While for interaction type, the form of equation is

where $u^{i}$ are determined and the initial distribution $\mu_{0}=\frac{1}{N}\sum_{i=1}^{N}\delta_{u^{i}}$ is known. Let $N\to\infty$, then the limit condition of the above two equations are

and

We see that for SDEs with interaction, one can determine a stochastic flow $\{x(\cdot,t),t\geq 0\}$, whose evolution depends on the measure transferred by the flow. This measure can be treated as a mass distribution of the interacting particle system. This is the main difference with Mckean-Vlasov type.

Then it is natural to consider controlling the system, whose equation contains interaction. For example, we control the trajectory of each particles in order to get their mass distribution close to a given distribution $\nu$ and hope to use less energy of control at the same time. So that we may set the state function has the form of (2) and the cost function is

where $A_{t}$ is a determined positive function and $\rho$ is a distance between measures, for example, defined as (4).

For the optimal control problem, the adjoint process is used to get the necessary condition for optimal control (known as Pontryagin maximum principle) [11, 12, 13, 14], some related works refer to [15, 16, 17, 18, 19, 20]. More precisely, for the classical stochastic optimal control problem, in which coefficients do not depend on the distribution of masses, the adjoint process is the solution $(Y,Z)$ to a backward stochastic differential equation (BSDE) [21, 22, 23], which has the following form:

In our case the state equation (2) depends on the distribution of masses of the system $\mu_{t}$, as well as functions $f$ and $g$ in the cost function (3). This leads us to the necessity to use a derivative of a function with respect to a probability measure in order to define an adjoint process to the optimal control problem with interactions. Using the existence and uniqueness of the generalized BSDE we will get an analog of Pontryagin stochastic maximum principle for optimal control of the equation with interaction.

In this paper, we will construct an adjoint process in order to get the condition that the optimal control should satisfy. We will get the analogue of Pontryagin stochastic maximum principle for the optimal control problem (2-3). To this aim, we introduce a generalized backward stochastic differential equation with interaction and prove the existence and uniqueness of the solution. Note that one form of backward stochastic differential equation was considered in [24]. But in order to apply it to a control problem with interaction, we should have BSDE with a more general form. The main difficulty that arises here is that in such BSDE with interaction, the coefficients of the equation depend on a random distribution of masses of the flow. The statement of existence and uniqueness of the solution is interesting itself.

The rest of this paper has the following structure. In Section 2, we recall the notion of L-differentiability for functions of probability law. We will use this type of differentiability to introduce an adjoint equation to a control problem with interaction. We define the generalized backward stochastic differential equation with interaction in Section 3. Here we prove the existence and uniqueness of the solution to this equation. Section 4 is devoted to the control problem with interaction. The corresponding adjoint equation is given in the form of a backward stochastic differential equation with interaction from Section 3. We end this section with the maximum principle for optimal control. Some examples are considered in Section 5 to illustrate the main results.

As we mentioned in the Introduction, in order to define the adjoint process to the stochastic optimal control for the equation with interactions, we need to differentiate the coefficients of equations with respect to probability measure. There are many approaches for the definition of differentiability of a real-valued function that is defined on a space of measures (see, for example, [25, 26, 27, 28] ). We will use the concept of differentiability in the sense of P.L.Lions for functions of measure (see, for example, [29]). Let $L^{p}(\mathbb{R}^{d},\mathcal{B},\mu;\mathbb{R}^{d})$ be the set of $\mathcal{B}$ measurable variables $X:\mathbb{R}^{d}\to\mathbb{R}^{d}$ such that $||X||_{L^{p}}=\left(\int_{\mathbb{R}^{d}}|x|^{p}\mu(dx)\right)^{\frac{1}{p}}<\infty$. Let $\mathcal{P}(\mathbb{R}^{d})$ be the set of mass measures $\mu$ on $(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}))$, and

The 2-Wasserstein metric for $\mu^{1},\mu^{2}\in\mathcal{P}_{2}(\mathbb{R}^{d})$ defined as

For any function $h:\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}$, we let ”lifted” function $\tilde{h}:L^{2}(\mathcal{B};\mathbb{R}^{d})\to\mathbb{R}^{d}$ defined by $\tilde{h}(X)=h(\mu_{X}),$ where $X\in L^{2}(\mathcal{B};\mathbb{R}^{d})$ is a random variable with distribution $\mu.$ If for $\mu\in\mathcal{P}_{2}(\mathbb{R}^{d})$, there exists $X\in L^{2}(\mathcal{B};\mathbb{R}^{d})$ such that $\mu=\mu_{X}$ and $\tilde{h}:L^{2}(\mathcal{B};\mathbb{R}^{d})\to\mathbb{R}^{d}$ is Fréchet differentiable at $X$, then $h:\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}$ is said to be differentiable at $\mu$ and the derivative is defined as follows.  

Since $L^{2}(\mathcal{B};\mathbb{R}^{d})$ is $Hilbert$ space, due to the Riesz representation theorem, and as shown by Lions [30], there exists a Borel measurable function $g:\mathbb{R}^{d}\to\mathbb{R}^{d}$ such that

where $\rho$ is the adjoint probability measure of $X$ and $Y$, and $g$ depends on $X$ only through its law $\mu_{X}$. Thus, we can write (5) as

The function $g(\cdot)$ is called the derivative of $h:\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}$ at $\mu=\mu_{X}$ and it is denoted by $h_{\mu}(\mu,y)=g(y),\quad y\in\mathbb{R}^{d}$.

Consider the following generalized BSDE with interaction

where $\mu_{0}\in\mathcal{P}(\mathbb{R}^{d})$ is the given mass distribution, and for all $u,v\in\mathbb{R}^{d},$ $E|\xi(u_{1})-\xi(u_{2})|^{2}\leq L|u_{1}-u_{2}|^{2}$. $\Phi(u,t,v,y)$ and $\Psi(u,t,v,z)$ are measurable functions on $\mathbb{R}^{d}\times[0,T]\times\mathbb{R}^{d}\times\mathbb{R}^{d}$ and $\mathbb{R}^{d}\times[0,T]\times\mathbb{R}^{d}\times\mathbb{R}^{d\times m}$ with values in $\mathbb{R}^{d}$, respectively. The function $f(u,t,y,z,\mu,\nu)$ is $\mathcal{F}_{t}-$adapted on $\mathbb{R}^{d}\times[0,T]\times\mathbb{R}^{d}\times\mathbb{R}^{d\times m}\times\mathcal{P}(\mathbb{R}^{d})\times\mathcal{P}(\mathbb{R}^{d})$ with values in $\mathbb{R}^{d}$.

In order to formulate the definition of the solution to this generalized BSDE with interaction we introduce the following notations. Let $\mu_{0}$ is a given mass measure.

$\mathbb{H}_{T}^{2}\left(\mathbb{R}^{d}\right)$, the space of all predictable processes $\varphi:\Omega\times\mathbb{R}^{d}\times[0,T]\mapsto\mathbb{R}^{d}$ such that $\|\varphi\|^{2}=$ $E\int_{\mathbb{R}^{d}}\int_{0}^{T}\left|\varphi_{t}\right|^{2}dt\mu_{0}(du)<+\infty.$

For $\beta>0$ $,\|\phi\|_{\beta}^{2}$ denotes $\mathbb{E}\int_{\mathbb{R}^{d}}\int_{0}^{T}e^{\beta t}\left|\phi(u,t)\right|^{2}dt\mu_{0}(du).$

$\mathbb{H}_{T,\beta}^{2}\left(\mathbb{R}^{d}\right)$ denotes the space of measurable functions $\phi:\Omega\times\mathbb{R}^{d}\times[0,T]\mapsto\mathbb{R}^{d}$ $\phi\in\mathbb{H}_{T}^{2}\left(\mathbb{R}^{d}\right)$ endowed with the norm $\|\cdot\|_{\beta}$.

Note that the solution of this type of BSDE is given by

where $M(u,t)=\mathbb{E}^{\mathcal{F}_{t}}\left[\xi(u)+\int_{0}^{T}f(u,s,y(u,s),z(u,s),\mathcal{M}^{y,u}_{s},\mathcal{M}^{z,u}_{s})dt\right]$ is a martingale and the function $z(u,t)$ is given by the martingale represent theorem

We assume that there are exist constants $L_{1},\ L_{2}>0$ such that for all $t\in[0,T],u,v\in\mathbb{R}^{d}$,

and

where $R_{1}^{2}=\left(|y_{1}-y_{2}|^{2}+|z_{1}-z_{2}|^{2}+W_{2}^{2}(\mu_{1},\mu_{2})+W_{2}^{2}(\nu_{1},\nu_{2})\right)$, $W_{2}(\cdot,\cdot)$ is the 2-Wasserstein metric and $R_{2}^{2}=\left(|v_{1}-v_{2}|^{2}+|x_{1}-x_{2}|^{2}\right)$.