An Optimal Control Perspective on
Classical and Quantum Physical Systems.

In the last two decades, there has been increasing interest among physicists in applying ideas from physics to finance and economics, as one can see in some classical texts.[1]  [2]  [3] In this paper, we want to do the exact opposite, that is, we apply ideas from the optimal control theory usually used in finance and economics,[4]   [5]  [6]  [7]  [8] to the classical and quantum physical world. Indeed, open and closed-loop control problems are not in the toolbox of mathematical methods in physics; nevertheless, dynamic optimization with its optimal control theory is the corner-stone of modern economic analysis.

There is not much literature associated with this endeavor. Recently, there have appeared some studies of how a control problem associated with an economic model can be interpreted as a second class constrained physical system.[9]  [10]  [11]  [12] In [9] it is found that at the classical level, the constrained dynamics given by Dirac’s brackets are the same as the dynamics given by the Pontryagin equations [13]. The right quantization of this second-class constrained system ended with a Schrödinger equation that is just the Hamilton–Jacobi–Bellman equation used in optimal control theory and economics [14]. It was surprising to find that after the quantization of the Pontryagin theorem, Bellman’s maximum principle is obtained.

From the perspective of control theory, this is an exciting finding in terms of the feedback of these systems. The Pontryagin theory is characterized by open-loop strategies for the Lagrangian multiplier, whereas the Hamilton–Jacobi–Bellman equation has closed-loop strategies intrinsically. Since Pontryagin’s theory is equivalent to a classical mechanical model and Bellman’s theory is just a quantum mechanical one, this induces one to relate open-loop strategies with classical mechanics and closed-loop strategies with quantum mechanics. All this sounds as if the quantizations of open-loop strategies would be the closed-loop ones.

Thus, if an economic feedback system characterized by an optimal control theory model can be seen as a physical system, one can naturally ask the inverse question: can the physical systems be seen from an optimal control perspective? Or, more specifically, do there exist open-loop and closed-loop strategies in physical systems? If the answer is affirmative, then, where are they? Moreover, how and why do these feedback problems appear? This paper gives some clues and answers to all these questions.

For this purpose, we found that the classical theory must be represented as a closed-loop feed-back problem. Indeed, due to the invariance of the classical Hamiltonian equations of motion (which represent an open-loop dynamics) by canonical transformations, both schemes are equivalent. In the same way, from the perspective of quantum physics, we see that if one identifies the momentum with the Lagrange multiplier, the relation between momentum and position is just a closed-loop strategy in optimal control theory. Thus, both open and closed-loop feed-backs are also a natural phenomenon in our physical world.

In the following three sections we review, for the non specialist reader, the ideas of closed and open-loop strategies that appear in control theory.

To emphasize the fundamental ideas and to keep the equations simple, a generic one-dimensional physical system will be considered. Generalizations to higher dimensions are straightforward. Consider an optimal control problem commonly used in financial applications.[4]  [5] It is required to optimize the cost functional

$$A[x,u]=\int_{t_{0}}^{t_{1}}F(x,u,t)\ dt,$$

(1)

where $x$ represents a state variable (for example, the production of a certain article) and $u$ is a control variable (such as the marketing costs). The state variable must satisfy the market dynamics

$$\dot{x}=f(x,u,t)\ \ \ \ \ \ x(t_{0})=x_{0}.$$

(2)

The problem is to determine how to obtain the production trajectory $x=x(t)$ and the control path $u=u(t)$ to optimize the cost functional. To get the solution, the method of Lagrange multipliers is applied, so an improved functional $A$ on the extended configuration space $(x,u,\lambda)$ is considered, which is defined by

$$A[x,u,\lambda]=\int_{t_{0}}^{t_{1}}F(x,u,t)-\lambda(\dot{x}-f(x,u,t))\ dt.$$

(3)

To obtain the solution, the integrand of (3) can be interpreted as the Lagrangian:

$$L(x,u,\lambda,\dot{x},\dot{u},\dot{\lambda})=F(x,u,t)-\lambda(\dot{x}-f(x,u,t)).$$

(4)

The extremal curves then satisfy the Euler–Lagrange equations:

$$\frac{\partial L}{\partial\lambda}-\frac{d}{dt}\ \big{(}\frac{\partial L}{\partial\dot{\lambda}}\big{)}=0,\ \ \ \ \ \ \ \frac{\partial L}{\partial x}-\frac{d}{dt}\ \big{(}\frac{\partial L}{\partial\dot{x}}\big{)}=0,\ \ \ \ \ \ \ \frac{\partial L}{\partial u}-\frac{d}{dt}\ \big{(}\frac{\partial L}{\partial\dot{u}}\big{)}=0.$$

(5)

These are also written respectively as

$$\dot{x}=\frac{\partial H}{\partial\lambda},\ \ \ \ \ \ \ \ \ \ \ \dot{\lambda}=-\frac{\partial H}{\partial x},\ \ \ \ \ \ \ \ \ \ \ \frac{\partial H}{\partial u}=0,$$

(6)

with $H$ defined by

$$H=H(x,u,\lambda)=F(x,u,t)+\lambda f(x,u,t).$$

(7)

Equations (6) are the well-known Pontryagin equations, which are obtained in optimal control theory, through the Pontryagin maximum principle, and give the solution to the optimization problem.

The action (3) can be written in a compact form as

$$A[x,u,\lambda]=\int_{t_{0}}^{t_{1}}-\lambda\dot{x}+H(x,u,\lambda,t)\ dt.$$

(8)

Strictly speaking, the Pontryagin equations must be obtained by optimizing the action (8) with respect to its three variables $x,u,\lambda$. If $\delta x,\ \delta u$ and $\delta\lambda$ are the corresponding functional variations of the initial variables, by expanding the Hamiltonian in a Taylor series, keeping the first-order terms only, integrating by parts, and using the fact that $\delta x(t_{0})=0$ (the initial point is fixed by the initial condition (2)), then one obtains

$$\delta A={\displaystyle\int_{t_{0}}^{t_{1}}\Bigl{[}\left(\frac{\partial H}{\partial\lambda}-\dot{x}\right)\delta\lambda+\left(\frac{\partial H}{\partial x}+\dot{\lambda}\right)\delta x+\frac{\partial H}{\partial u}\delta u\Bigr{]}dt-\lambda(t_{1})\delta x(t_{1}).}$$

(9)

To maximize the action, all the first-order terms in $\delta x,\delta u$ and $\delta\lambda$ must vanish. Now, it is well-known that there are two classes of control strategies:

• •

  open-loop strategies that depend only on time: $u=u(t)$, and

• •

  closed-loop strategies that depend on the state variable $x$ and time: $u=u(x,t)$. [15]

In the case of an open-loop strategy, the variables $x=x(t),u=u(t)$ and $\lambda=\lambda(t)$ are independent, and so $\delta x,\delta u$ and $\delta\lambda$ are linearly independent. Hence, the Pontryagin equations (6) and the transversality condition $\lambda(t_{1})=0$ can be obtained from Equation (9). In fact, the solutions of the Pontryagin equations give the extremal curves

$$x=x^{*}(t),\ \ \ \ \ \ \ \ \ \ \ \ \lambda=\lambda^{*}(t),\ \ \ \ \ \ \ \ \ \ \ \ u=u^{*}(t),$$

(10)

which are the solutions of Equations (6). But what happens, however, with closed-loop strategies? Here, due to the relations between the variables in $u=u(x,t)$, the functional variations are related by $\delta u=\frac{\partial u}{\partial x}\delta x$. Substituting this into (9) yields

$$\delta A={\displaystyle\int_{t_{0}}^{t_{1}}\Bigl{[}\left(\frac{\partial H}{\partial\lambda}-\dot{x}\right)\delta\lambda+\left(\frac{\partial H}{\partial x}+\frac{\partial H}{\partial u}\frac{\partial u}{\partial x}+\dot{\lambda}\right)\delta x\Bigr{]}dt-\lambda(t_{1})\delta x(t_{1}).}$$

(11)

If $\lambda$ and $x$ remain independent, the optimization of the functional implies

$$\frac{\partial H}{\partial\lambda}-\dot{x}=0\;,\qquad\quad\frac{\partial H}{\partial x}+\frac{\partial H}{\partial u}\frac{\partial u}{\partial x}+\dot{\lambda}=0,$$

(12)

plus the transversality condition, but the equation that gives the optimal condition for the control in (6) is lost. Then, if $u$ is not known as a fixed function of $x$ from the beginning, that is, one knows that the constraints $u=u(x,t)$ exist, but one does not know their specific form, one has three unknowns $x,u$, and $\lambda$ but only two equations of motion. Thus, the variational problem is inconsistent for an arbitrary closed-loop strategy $u=u(x,t)$ because the functional form of $u$ is not determined by the equations of motion. It must be given from the beginning.
Now, note that for a open-loop strategy, the control equation in (6) is, in fact, the following algebraic equation for $u$:

$$\frac{\partial H}{\partial u}=\frac{\partial F(x,u,t)}{\partial u}+\lambda\ \frac{\partial f(x,u,t)}{\partial u}=0.$$

(13)

From this equation, the optimal control $u^{*}$ can be obtained in principle, as a function of $x$ and $\lambda$:

$$u=u^{*}(x,\lambda,t),$$

(14)

One can conclude, then, that the same optimization problem implies that the naive optimal open-loop $u^{*}$ strategy in (10) is actually just a closed-loop strategy! How can this be consistent with the variational problem (9), in which $x$, $u$ and $\lambda$ are independent variables?
To understand this, consider an arbitrary closed-loop strategy of the general form $u=u(x,\lambda,t)$. Then  $\delta u=\frac{\partial u}{\partial x}\delta x+\frac{\partial u}{\partial\lambda}\delta\lambda$. After substituting this into (9), $\delta A$ equals

$${\displaystyle\int_{t_{0}}^{t_{1}}\Bigl{[}\left({\displaystyle\frac{\partial H}{\partial\lambda}+{\displaystyle\frac{\partial H}{\partial u}\frac{\partial u}{\partial\lambda}-\dot{x}}}\right)\delta\lambda+\left(\frac{\partial H}{\partial x}+\frac{\partial H}{\partial u}\frac{\partial u}{\partial x}+\dot{\lambda}\right)\delta x\Bigr{]}dt-\lambda(t_{1})\delta x(t_{1}).}$$

(15)

If $x(t)$ and $\lambda(t)$ are independent variables, we have the following equations of motion from the variational principle (15):

$$\frac{\partial H}{\partial\lambda}+\frac{\partial H}{\partial u}\frac{\partial u}{\partial\lambda}-\dot{x}=0,\ \ \ \ \ \ \ \ \ \ \frac{\partial H}{\partial x}+\frac{\partial H}{\partial u}\frac{\partial u}{\partial x}+\dot{\lambda}=0.$$

(16)

Obviously these equations, for an arbitrary closed-loop strategy $u=u(x,\lambda,t)$, differ from the Pontryagin open-loop equations. Now, choose $u$ as the special optimal closed-loop strategy $u^{*}$ which is the solution of (13), so Equations (16) are then reduced to

$$\frac{\partial H(x,u^{*}(x,\lambda,t),\lambda,t)}{\partial\lambda}-\dot{x}=0,\ \ \ \ \ \ \ \ \ \ \frac{\partial H(x,u^{*}(x,\lambda,t),\lambda,t)}{\partial x}+\dot{\lambda}=0.$$

(17)

But this last set of two equations is equivalent to the three Pontryagin equations. The solutions of (17) give optimal paths $x=x^{*}(t)$ and $\lambda=\lambda^{*}(t)$, from which the optimal control open-loop strategy $u^{*}(t)$ in (10) can be computed given the optimal closed-loop strategy (14), by

$$u=u^{*}(t)=u^{*}(x^{*}(t),\lambda^{*}(t),t)$$

(18)

Thus, the special optimal closed-loop strategy $u=u^{*}(x,\lambda,t)$ is completely equivalent to an open-loop strategy $u=u^{*}(t)$ over the optimal $x^{*}(t)$ and $\lambda^{*}(t)$ trajectories. Then, the optimal closed-loop strategy $u^{*}(x(t),\lambda(t),t)$ is the same object given by the open-loop one from a dynamical point of view. So from now on, we will not distinguish between them for the case of Pontryagin’s theory.
Note that for an arbitrary given closed-loop strategy $u=u(x,\lambda,t)$ which is not optimal ($\frac{\partial H}{\partial u}\neq 0$), Equations (16) mean that the action is optimized, but these extremals are not necessarily global. In fact, its dynamics is not given by Pontryagin’s equations. The optimal condition $\frac{\partial H}{\partial u}=0$ in (6) gives the global extremal for the action $A$, as was established by Pontryagin’s maximum principle.

To end this section, suppose that $\lambda$ and $x$ are not independent but are related by $\lambda=\lambda(x,t)$; then, the variation of $\lambda$ is $\delta\lambda=\frac{\partial\lambda}{\partial x}\ \delta x$. Substituting this into (15), choosing the optimal control strategy $u=u^{*}$, using the transversality condition $\lambda(t_{1})=0$ and the fact that $\dot{\lambda}={\displaystyle\frac{\partial\lambda}{\partial x}\dot{x}+\frac{\partial\lambda}{\partial t}}$, we get

$$\delta A=\int_{t_{0}}^{t_{1}}\Bigl{[}\left(\frac{\partial H}{\partial\lambda}\frac{\partial\lambda}{\partial x}-\dot{x}\frac{\partial\lambda}{\partial x}\right)+\left(\frac{\partial H}{\partial x}+\frac{\partial\lambda}{\partial x}\dot{x}+\frac{\partial\lambda}{\partial t}\right)\Bigr{]}\delta x\ \ dt,$$

(19)

but, as $H=F(x,u^{*},t)+\lambda f(x,u^{*},t)$, it follows that

$$\frac{\partial H}{\partial\lambda}=f(x,u^{*},t,)\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \frac{\partial H}{\partial x}=\frac{\partial F(x,u^{*},t)}{\partial x}+\lambda\frac{\partial f(x,u^{*},t)}{\partial x}.$$

Thus, (19) gives finally

$$\delta A=\int_{t_{0}}^{t_{1}}\Bigl{[}\left(f\frac{\partial\lambda}{\partial x}+\frac{\partial F}{\partial x}+\lambda\frac{\partial f}{\partial x}\right)+\frac{\partial\lambda}{\partial t}\Bigr{]}\delta x\ \ dt$$

or

$$\delta A=\int_{t_{0}}^{t_{1}}\Bigl{[}\frac{dH^{*}}{dx}+\frac{\partial\lambda}{\partial t}\Bigr{]}\delta x\ \ dt.$$

where

$$H^{*}=H^{*}(x,t)=H(x,u^{*}(x,\lambda(x,t),t),\lambda(x,t),t)$$

(20)

is the reduced Hamiltonian in terms of $x$. In this way, the optimization of the action $A$ implies that the closed-loop $\lambda=\lambda(x,t)$ strategy must satisfy the optimal consistency condition

$$\frac{dH^{*}(x,t)}{dx}+\frac{\partial\lambda}{\partial t}=0.$$

(21)

Equation (21) is closely related to the Hamilton–Jacobi–Bellman equation. In fact, if the Lagrangian multiplier is of the form $\lambda(x,t)=\frac{\partial J(x,t)}{\partial x}$, then Equation (21) implies that $J(x,t)$ satisfies the Hamilton–Jacobi–Bellman equation.[9]
Let $\lambda^{*}(x,t)$ be a solution of (21); then, the optimal state variable $x(t)$ can be obtained from (2), by

$$\dot{x}=f(x,u^{*}(x,\lambda^{*}(x,t),t),t).$$

(22)

Note that (22) can be seen as the Pontryagin equation for $x(t)$, in the sense that $u^{*}$ and $\lambda^{*}$ are chosen in such a way that they are optimal closed-loop strategies, that is, these strategies maximize or minimize the action.
Thus, there exist three types of strategies in optimal control theory: open-loop ($x=x(t),\lambda=\lambda(t),u=u^{*}(t)$), inert closed-loop ($x=x(t),\lambda=\lambda(t),u=u^{*}(x(t),\lambda(t),t)$) and $\lambda$ closed-loop ($x=x(t),\lambda=\lambda(x(t),t),u=u^{*}(x(t),\lambda(x(t),t),t)$) strategies. The first two are completely equivalent because these give the same dynamical equations.

A second approach to the optimization problem comes from dynamic programming theory, and was developed by Richard Bellman [14]. In this case, the fundamental variable is the optimal value of the action, defined by

$$J(x_{0},t_{0})=\max_{u}\Bigg{(}\int_{t_{0}}^{t}F(x,u,t)\ dt\Bigg{)},$$

(23)

subject to (2), with initial condition $x(t_{0})=x_{0}$.
The optimality principle of Bellman implies that $J(x,t)$ satisfies the Hamilton–Jacobi–Bellman equation[4]

$$\max_{u}\Bigl{(}F(x,u,t)+\frac{\partial J(x,t)}{\partial x}f(x,u,t)\Bigr{)}=-\frac{\partial J(x,t)}{\partial t}.$$

(24)

The left-hand side of Equation (24) is just the maximization of the Hamiltonian (13) with respect to the control variable $u$, where the Lagrangian multiplier $\lambda$ of the Pontryagin approach must be identified with $\frac{\partial J(x,t)}{\partial x}$. Thus, the Bellman theory is (from the Pontryagin perspective) a model which has a closed-loop $\lambda$-strategy $\lambda(x,t)=\frac{\partial J(x,t)}{\partial x}$.
By maximizing and solving for the optimal control variable in the left-hand side of (24) as $u^{*}=u^{*}(x,t)=u^{*}(x,\lambda(x,t),t)=u^{*}(x,\frac{\partial J(x,t)}{\partial x},t)$, the Hamilton–Jacobi–Bellman equation is

$$F(x,u^{*},t)+\frac{\partial J(x,t)}{\partial x}f(x,u^{*},t)=-\frac{\partial J(x,t)}{\partial t}.$$

(25)

Differentiating in (25) with respect to $x$, one gets

$$\begin{array}[]{l}{\displaystyle\frac{\partial F(x,u^{*},t)}{\partial x}+{\displaystyle\frac{\partial^{2}J(x,t)}{\partial x^{2}}f(x,u^{*},t)+\frac{\partial J(x,t)}{\partial x}\frac{\partial f(x,u^{*},t)}{\partial x}}}\\ \\ +\Bigg{(}{\displaystyle\frac{\partial F(x,u^{*},t)}{\partial u^{*}}+{\displaystyle\frac{\partial J(x,t)}{\partial x}\frac{\partial f(x,u^{*},t)}{\partial u^{*}}\Bigg{)}\frac{du^{*}(x,t)}{dx}}}\\ \\ =-{\displaystyle\frac{\partial^{2}J(x,t)}{\partial x\partial t}.}\end{array}$$

Using the fact that $u^{*}$ is optimal and replacing $\lambda(x,t)=\frac{\partial J(x,t)}{\partial x}$, one obtains

$${\displaystyle\frac{\partial F(x,u^{*},t)}{\partial x}+\frac{\partial\lambda(x,t)}{\partial x}f(x,u^{*},t)+\lambda\frac{\partial f(x,u^{*},t)}{\partial x}=-{\displaystyle\frac{\partial\lambda(x,t)}{\partial t},}}$$

or

$$\frac{dH^{*}(x,t)}{dx}+\frac{\partial\lambda(x,t)}{\partial t}=0.$$

This equation is identical to (21). Thus, this optimal consistency condition for the closed-loop $\lambda^{*}$-strategy is nothing more than the derivative of the Hamilton–Jacobi–Bellman equation. Then, Equation (21) can be written, according to (25), as

$$\frac{d}{dx}\Bigg{(}F(x,u^{*},t)+\frac{\partial J(x,t)}{\partial x}f(x,u^{*},t)+\frac{\partial J(x,t)}{\partial t}\Bigg{)}=0.$$

(26)

Integrating the above equation gives finally

$$F(x,u^{*},t)+\frac{\partial J(x,t)}{\partial x}f(x,u^{*},t)+\frac{\partial J(x,t)}{\partial t}=g(t),$$

where $g(t)$ is an arbitrary, time-dependent function.
Then, for an optimal closed-loop $\lambda^{*}$-strategy, the optimization problem gives a non-homogeneous Hamilton–Jacobi–Bellman equation (if one identifies the Lagrangian multiplier with $\frac{\partial J(x,t)}{\partial x}$). The Bellman maximum principle instead gives a homogeneous Hamilton–Jacobi–Bellman equation.

The classical equation of motion of a generic physical system can be obtained from the Hamiltonian action $A[x,p_{x}]$ defined by

$$A[x,p_{x}]=\int p_{x}\dot{x}-H(x,p_{x})\ dt,$$

(27)

(where $H$ is the Hamiltonian function) as an optimization problem. In this case, by expanding the Hamiltonian in a Taylor series, keeping the first-order terms only and integrating by parts, one has that the variation of the action is

$$\delta A={\displaystyle\int_{t_{0}}^{t_{1}}\Bigl{[}\left(\frac{\partial H}{\partial p_{x}}-\dot{x}\right)\delta p_{x}+\left(\frac{\partial H}{\partial x}+\dot{p_{x}}\right)\delta x\Bigr{]}dt+\bigl{[}p_{x}(t_{2})\delta x(t_{2})-p_{x}(t_{1})\delta x(t_{1})\bigr{]}}$$

(28)

To maximize the action, all the first-order terms in $\delta x$ and $\delta p_{x}$ must vanish. For this, it is supposed that:

• •

  the end points of the curve $x(t)$ are fixed, so $\delta x(t_{2})=0$ and $\delta x(t_{1})=0$, and

• •

  the variables $x$ and $p_{x}$ are considered independent,

so $\delta x$ and $\delta p_{x}$ are linearly independent, from which the Hamiltonian equations of motion are obtained from (28) as:

$$\dot{x}=\frac{\partial H(x,p_{x})}{\partial p_{x}},\ \ \ \ \ \ \ \ \ \ \dot{p}_{x}=-\frac{\partial H(x,p_{x})}{\partial x}.$$

(29)

From an economic or dynamic optimization point of view, the problem of optimizing the action (27) is analogous to an optimal control problem, but without a control variable $u$. As we have seen in the previous section, the solutions of the control problem are given by the Pontryagin equations (6) plus the transversality condition. Note that the first two Pontryagin equations in (6) are precisely the Hamiltonian equations of motion (29) if one identifies the Lagrangian multiplier $\lambda$ with the canonical momentum $p_{x}$. This implies that $\lambda$ corresponds to a $\lambda$ open-loop strategy for the system ($x(t)$ and $\lambda(t)=p_{x}(t)$ are independent), which is consistent with the Hamiltonian equations of motions. Thus, the Hamiltonian theory is a open-loop model similar to Pontryagin’s theory.

Actually one can write $p$ in terms of $x$ from (30) by solving the constraint:

$\displaystyle p_{x}=p_{x}(x,t).$

(31)

Using the analogy of the momentum $p$ with the Lagrangian multiplier $\lambda$, Equation (31) corresponds to the closed-loop $\lambda$-strategy from an economic point of view. Thus $x$ and $p$ are not independent in this case and their variations are related by $\delta p_{x}=\frac{\partial p_{x}}{\partial x}\delta x$. By replacing in (28) and using the fact that the endpoints of $x$ are fixed, one arrives at

$$\delta A={\displaystyle\int_{t_{0}}^{t_{1}}\Bigl{[}-\frac{\partial p_{x}}{\partial t}-\frac{\partial H}{\partial x}-\frac{\partial H}{\partial p_{x}}\frac{\partial p_{x}}{\partial x}\Bigr{]}\delta x\ dt.}$$

(32)

By defining the reduced Hamiltonian $H^{*}(x,t)=H(x,p_{x}(x,t),t)$, Equation (32) can be written as

$$\delta A={\displaystyle\int_{t_{0}}^{t_{1}}\Bigl{[}-\frac{\partial p_{x}}{\partial t}-\frac{\partial H^{*}}{\partial x}\Bigr{]}\delta x\ dt.}$$

(33)

The optimization of the action then gives

$$-\frac{\partial p_{x}}{\partial t}-\frac{\partial H^{*}}{\partial x}=0.$$

(34)

This last equation is a consistency condition that the closed-loop strategy (31) must satisfy, to give an extremal of the action. That is, if $p_{x}=p_{x}(x,t)$ satisfies (34), then $p_{x}=p_{x}(x,t)$ is an optimal closed-loop strategy.
Now, if the closed-loop momentum strategy $p_{x}=p_{x}(x,t)$ is just the derivative of some function $S(x,t)$, such as

$\displaystyle p_{x}(x,t)=\frac{\partial S(x,t)}{\partial x},$

(35)

condition (34) gives

$$\frac{\partial}{\partial x}\Bigl{[}-\frac{\partial S(x,t)}{\partial t}-H^{*}(x,t)\Bigr{]}=0.$$

(36)

So, one obtains, by integration, that

$$-\frac{\partial S(x,t)}{\partial t}-H(x,\frac{\partial S(x,t)}{\partial x})=g(t)$$

(37)

for some function $g$ of time. The above equation is just an inhomogeneous Hamilton–Jacobi equation. Thus, the derivative of the Hamilton–Jacobi equation can be seen as the consistency condition to give to the action an extremal in the closed-loop $p_{x}$-strategy case. Also, this little analysis implies that closed-loop momentum strategies are closely related to the Hamilton–Jacobi equation.
Now, it is well known that the Hamilton–Jacobi equation appears in classical mechanics in the context of the canonical transformations, but how and where do the closed-loop $p_{x}=p_{x}(x,t)$ strategies appear there? In the next section, a short review of canonical transformations will be given and it will be elucidated how the closed-loop strategies appear in that context.

Thus, the “two observers” perspective of classical mechanics, through the method of canonical transformation, is responsible for the generation of the closed-loop $p_{x}$-strategies.
From (49) it is not clear if the $p_{x}$ closed-loops strategies thus generated satisfy the consistency condition (34) or if they satisfy the second condition (35) to obtain a Hamilton–Jacobi equation as in (37) for $S$. Instead, one can see these constant coordinate lines in terms of Equations (43) and (44). In fact, these equations are equivalent to (49), because the canonical transformation can be reconstructed from (43) and (44).
A constant $P_{0}$-line in (49) is equivalent (from (43)) to

$$p_{x}=\frac{\partial S(x,P_{0},t)}{\partial x},$$

(50)

thus, the constant $P_{0}$-line in (49) satisfies (35). From (45) and (46), $S(x,t)$ satisfies the Hamilton–Jacobi equation as in (37) with $g(t)=0$. This implies that a closed-loop strategy generated by (50) satisfies the consistency relation (34), so (50) defines a true optimal closed-loop $p_{x}$-strategy.
For the $Q_{0}$ constant coordinate line in (49) one has however due to (44) that

$$Q_{0}=\frac{\partial S(x,P,t)}{\partial P},$$

(51)

but from this equation one cannot obtain $p_{x}$ in terms of $x$, so this line does not generates a $p_{x}$ closed-loop strategy at all. This is due to the structure of Equations (43) and (44). In order to reconstruct the canonical transformation (38) it is required to invert the system (43), (44). Note that only from (43) can the momentum $P$ be written in terms of the first-observer variables as $P=P(x,p_{x},t)$. But from (44) alone one cannot solve $Q$ in terms of the $x,p_{x}$. The other equation (43) is needed to do that. Thus, a constant $Q_{0}$-line alone can not generate a true closed-loop $p_{x}=p_{x}(x,t)$ strategy.

In this way, closed-loop $p_{x}$-strategies appear in classical mechanics as a consequence of the two observers interpretation of the canonical transformation theory. These closed-loop strategies are inert in the same way that the optimal $u^{*}$ closed-loop ones are inert in control theory. This is because both closed-loop approaches $p_{x}$ and $u^{*}$ gives the same dynamical equation of the open-loop case. For the closed-loop $p_{x}$ case, the open-loop dynamics analogous to the given by Pontryagin’s equations are just provided by the Hamiltonian equations of motion of the first observer in the $(x,p_{x})$ phase space.

In this section, the origins of the Hamilton–Jacobi–Bellman equation that appears in the limit $\hbar\rightarrow 0$ in the quantum phenomena, will be explained as a consequence of the emergence of closed-loop $p_{x}$-strategies in the quantum world.
Consider the Schrödinger equation for a non-relativistic particle of mass $m$:

$$-\frac{\hbar^{2}}{2m}\frac{\partial^{2}\Psi(x,t)}{\partial x^{2}}+U(x)\Psi(x,t)=i\hbar\frac{\partial\Psi(x,t)}{\partial t}.$$

(52)

Writing the wave function in the form

$$\Psi(x,t)=e^{\frac{i}{\hbar}S(x,t)},$$

(53)

and by substituting (53) into the Schrödinger equation, the following equation Quantum Hamilton–Jacobi equation for $S(x,t)$ is obtained:

$$\frac{1}{2m}\Bigl{(}\frac{\partial S(x,t)}{\partial x}\Bigr{)}^{2}+U(x)-\frac{i\hbar}{2m}\frac{\partial^{2}S(x,t)}{\partial x^{2}}=-\frac{\partial S(x,t)}{\partial t}.$$

(54)

Note that this equation is completely equivalent to Schrödinger’s equation, but here, the classical and quantum realms can be clearly identified. In fact, by taking the limit $\hbar\rightarrow 0$ in (54), one gets

$$\frac{1}{2m}\Bigl{(}\frac{\partial S(x,t)}{\partial x}\Bigr{)}^{2}+U(x)=-\frac{\partial S(x,t)}{\partial t}.$$

(55)

Equation (55) is just the classical Hamilton–Jacobi equation

$$H(x,\frac{\partial S(x,t)}{\partial x})=-\frac{\partial S(x,t)}{\partial t},$$

(56)

for the classical Hamiltonian function associated to the non-relativistic particle

$$H(x,p_{x})=\frac{p_{x}^{2}}{2m}+U(x),$$

(57)

where one must identify $p_{x}$ with the derivative of $S$

$$p_{x}=p_{x}(x,t)=\frac{\partial S(x,t)}{\partial x}.$$

(58)

to make contact with the classical Hamiltonian theory. And it is precisely this identification which generates the closed-loop $p_{x}$-strategy through (58). Note that it is induced from the quantum realm to the classical world, in the limit $\hbar\rightarrow 0$, through the classical Hamilton–Jacobi equation. The identification in (58) is a pure quantum phenomenon. In fact, considering the momentum operator

$$\hat{p}_{x}=-i\hbar\frac{\partial}{\partial x},$$

(59)

this operator is characterized by its eigenfunctions and eigenvalues:

$$\hat{p}_{x}\Phi_{p_{x}}(x)=p_{x}\Phi_{p_{x}}(x).$$

(60)

where the solution of this equation gives

$$\Phi_{p_{x}}(x)=e^{\frac{i}{\hbar}p_{x}x}.$$

(61)

In this context, $p_{x}$ and $x$ are independent variables. In fact the eigenfunction (61) corresponds to states with well defined values of the momentum.
Note now that if one applies the momentum operator to a generic wave function $\Psi$ which is a solution of the Schrödinger equation (written in the “momentum form” (53)), one obtains

$$\hat{p}_{x}\Psi(x,t)=\frac{\partial S(x,t)}{\partial x}\Psi(x,t).$$

(62)

By looking at the wave function as a vector with a continuous index $x$, the above equation implies that (locally at each point $x$) the momentum operator is diagonal, so that any wave function can be seen as an eigenstate of the momentum operator with momentum eigenvalue $\frac{\partial S(x,t)}{\partial x}$. Thus, one must identify the momentum eigenvalue $p_{x}$ in this quantum state with the derivative of the $S$ function through (58). It is just this identification which generates the closed-loop $p_{x}$-strategies directly in the quantum world.
On the other hand, the same Heisenberg canonical commutation relations

$$[\check{x},\check{P_{x}}]=\check{x}\check{P_{x}}-\check{P_{x}}\check{x}=i\hbar\ \check{I}.$$

(63)

can be seen as a constraint in the non-commutative phase space $(\check{x},\check{P_{x}})$. Thus, from (63) one could “solve” the momentum operator $\check{P_{x}}$ in terms of the $\check{x}$ operator. This necessarily implies the existence of a certain relation between $\check{P_{x}}$ and $\check{x}$ or between their eigenvalues. The representation of the canonical operator as a differential operator acting on a function space or Hilbert space as

$$\check{x}\rightarrow x,\ \ \ \ \ \hat{p}_{x}\rightarrow-i\hbar\frac{\partial}{\partial x}$$

(64)

is equivalent to solving the constraint (63), because on any wave function $\Psi(x,t)$, Equation (63) is satisfied identically. The memory of the quantum constraint (63) is then transferred in a local way to the momentum eigenvalue, according to (62). In a sense, the representation of the wave function as $\Psi(x,t)=e^{\frac{i}{\hbar}S(x,t)}$ locally diagonalizes the momentum operator $\check{p}_{x}$ over any quantum state, and (54) is just the Schrödinger equation in this diagonal basis. Note that all this is a kinematic effect created by the Heisenberg commutation relation (63); the dynamical effects appear when the explicit form of $S(x,t)$ is needed, and for that, one must solve the full Quantum Hamilton–Jacobi equation (54) explicitly. Note that Quantum Mechanics, as in (63), can be viewed as a constrained system in a non-commutative space, so, one would apply a generalization of Dirac’s method [18]  [19]  [20]  [21] to non-commutative spaces [22] to study quantum mechanical systems.

We can say, then, that closed-loop $p_{x}$-strategies correspond to a pure quantum phenomenon and are a consequence of Heisenberg’s uncertainty principle. In an arbitrary quantum state, momentum and position cannot be independent: they are related through the non-commutative character of the position and momentum operators. In a more defined momentum state, a less defined position state would emerge. Thus, these two variables must depend on one another in some way. Relation (58) is tantamount to a conversation between them. Only in a pure-momentum state, as given in (61), does the link disappear and position and momentum become independent variables. In fact, in a pure-momentum state, $p_{x}(x,t)=p_{x}^{0}$ is constant, that is: all the eigenvalues are the same, the value $x$ of the position (so its matrix is a multiple of the identity matrix), so Equation (58) gives

$\displaystyle S(x,t)=p_{x}^{0}x+\phi(t)$

(65)

as a solution, where $\phi(t)$ is some function of time. Thus, the wave function is

$$\Psi(x,t)=e^{\frac{i}{\hbar}(p_{x}^{0}x+\phi(t))}=e^{\frac{i}{\hbar}\phi(t)}\Phi_{p_{x}^{0}}(x)$$

(66)

which is the same momentum eigenstate amplified by a temporal arbitrary phase. Then the linear character of $S(x,t)$ in terms of $x$ implies that $p_{x}$ and $x$ are independent variables, and no closed-loop $p_{x}$-strategy exists in this case.
The same can be said for a pure-position eigenstate. Thus, closed-loop $p_{x}$-strategies are an inherent part of the quantum mechanical world and permeate the classical world in the limit $\hbar\rightarrow 0$ through the Hamilton–Jacobi equation. In the following two sections, we analyze some common textbook examples from closed-loop strategies’ point of view.

The quantum and classical Hamilton–Jacobi equations (54) and (55) are non-stationary equations, that is, they depend explicitly on time. In Quantum Mechanics, stationary states play a fundamental role. They are defined by

$$\Psi(x,t)=e^{-\frac{i}{\hbar}Et}\Phi(x).$$

(67)

By substituting this into the time-dependent Schrödinger equation (52), the time independent or stationary Schrödinger equation is obtained:

$$-\frac{\hbar^{2}}{2m}\frac{\partial^{2}\Psi(x)}{\partial x^{2}}+U(x)\Psi(x)=E\Psi(x).$$

(68)

Now by writing

$$\Psi(x)=e^{\frac{i}{\hbar}W(x)}$$

(69)

and substituting into (68) the stationary Quantum Hamilton–Jacobi equation holds:

$$\frac{1}{2m}\Bigl{(}\frac{\partial W(x)}{\partial x}\Bigr{)}^{2}+U(x)-\frac{i\hbar}{2m}\frac{\partial^{2}W(x)}{\partial x^{2}}=E.$$

(70)

Taking again the classical limit $\hbar\rightarrow 0$ in (70) the stationary Classical Hamilton–Jacobi equation appears:

$$\frac{1}{2m}\Bigl{(}\frac{\partial W(x)}{\partial x}\Bigr{)}^{2}+U(x)=E.$$

(71)

Due to (67) and (69), we have $\Psi(x,t)=e^{\frac{i}{\hbar}S(x,t)}=e^{\frac{i}{\hbar}(W(x)-Et)}$ which implies that

$$S(x,t)=W(x)-Et$$

(72)

for the stationary case. In this case, the closed-loop $p_{x}$-strategies are given by $p_{x}=p_{x}(x,t)=\frac{\partial W(x)}{\partial x}$.

But what about the non-stationary closed-loop $p_{x}$-strategies in the classical limit? In order to analyze this case, consider the example of a free particle, that is, $U(x)=0$. The non-stationary classical Hamilton–Jacobi equation is now

$$\frac{1}{2m}\Bigl{(}\frac{\partial S(x,t)}{\partial x}\Bigr{)}^{2}=-\frac{\partial S(x,t)}{\partial t}.$$

(73)

One can find a solution of the form  $S(x,t)=\frac{1}{2}a(t)x^{2}$,  so by substituting in (73) one gets $a(t)=\frac{-1}{(P_{0}-\frac{t}{m})}$, so $S(x,t)=\frac{1}{2}\frac{-x^{2}}{(P_{0}-\frac{t}{m})}$, and the corresponding closed-loop $p_{x}$-strategy is

$$p_{x}(x,t)=\frac{\partial S(x,t)}{\partial x}=\frac{-x}{(P_{0}-\frac{t}{m})}.$$

(74)

One can evaluate $x(t)$ using Equation (51)

$$Q_{0}=\frac{\partial S(x,P_{0},t)}{\partial P_{0}},$$

(75)

where the integration constant $P_{0}$ must be identified with the constant momentum for the second observer in the coordinate system $(Q,P)$. Thus

$$Q_{0}=\frac{1}{2}\frac{x^{2}}{(P_{0}-\frac{t}{m})^{2}},$$

(76)

from which $x(t)$ is computed as

$$x(t)=\sqrt{2\left(P_{0}-\frac{t}{m}\right)^{2}Q_{0}}=\sqrt{2Q_{0}}\left(P_{0}-\frac{t}{m}\right).$$

(77)

The associated open-loop $p_{x}(t)$ strategy is found by $p_{x}(t)=p_{x}(x(t),t)$ similarly to Equation (18). Thus, by substituting (77) into (74):

$$p_{x}(t)=\frac{-x(t)}{(P_{0}-\frac{t}{m})}=\frac{-\sqrt{2Q_{0}}(P_{0}-\frac{t}{m})}{(P_{0}-\frac{t}{m})}=-\sqrt{2Q_{0}}=p_{0},$$

(78)

so

$$x(t)=-p_{0}\left(P_{0}-\frac{t}{m}\right)=p_{0}\frac{t}{m}+x_{0},$$

(79)

where $x_{0}=-P_{0}p_{0}$. These last equations are the solutions for the motion of a free particle of course!
Now from the Hamiltonian equations, one gets the open-loop dynamics for the free particle:

$$\dot{x}=\frac{p_{x}}{m},\ \ \ \ \ \ \ \ \ \dot{p}_{x}=-\frac{\partial U(x)}{\partial x}=0,$$

(80)

so the open-loops dynamics is

$$x(t)=\frac{p_{0}}{m}t+x_{0},\ \ \ \ \ \ \ \ \ p_{x}(t)=p_{0}.$$

(81)

Then, the open-loop $p_{x}$-strategy for the free particle coming from the Hamiltonian equations of motions is identical to the non-stationary closed-loop $p_{x}$-strategy coming from the non-stationary classical Hamilton–Jacobi equation. In fact, any solution of the Classical Hamilton–Jacobi equation would be an origin for a (stationary and non-stationary) closed-loop $p_{x}$-strategy, and this one has to be equivalent with the open-loop strategy coming from the Hamiltonian equations of motion. This is because the Classical Hamilton–Jacobi equation corresponds to a “two observer” point of view of classical mechanics. The function $S(x,t)$ is just the generator of the canonical transformation which leaves the Hamiltonian equations invariant. Thus, the two schemes,

1. 1.

   the Hamiltonian “one observer” approach with its open-loop $p_{x}$-strategies and

2. 2.

   the Classical Hamilton–Jacobi “two observer” approach with its closed-loop $p_{x}$-strategies

are equivalent, because they have the same equation of motion constructed through the canonical transformation. Closed-loop strategies coming from the Hamilton–Jacobi equation are similar in character to the inert optimal $u^{*}(x,t)$ closed-loop strategies of the Pontryagin approach to optimal control theory, because they are equivalent to the open-loop ones.