Does Optimal Control Always Benefit from Better Prediction? An Analysis Framework for Predictive Optimal Control

We consider the following discrete-time dynamic system

where $x\in\mathbb{R}^{d_{n}}$, $w\in\mathbb{R}^{d_{l}}$, and $u\in\mathbb{R}^{d_{m}}$. The state $x_{k}$ and the generalized disturbance $w_{k}$ can be directly measured at step $k$. $w$ is usually considered as the output of a complex system, which is called the environment. $u_{k}$ is the control input.

This dynamic system represents the physical system to be controlled, such as a vehicle, a robot, a machine, or a building. The measured generalized disturbance $w$ may be the human input such as the pedal position and steering of the vehicle, or the ambient factors such as surrounding traffic behavior or the temperature. We assume that $w$ has a relatively large impact on the dynamics and the cost. Therefore, when we design the control $u$, we want to consider the impact of $w$.

We consider a finite-horizon problem of $N$ steps. The cost $J$ in this finite horizon is the sum of running costs from step 0 to step $N-1$ and the terminal cost,

The goal is to find a policy $u=\pi(\cdot)$ to minimize the cost $J$.

If the complete disturbance sequence $\bar{w}=[w_{0},w_{1},...,w_{N-1}]$ (the brackets here mean an ordered sequence) is known at step 0, the problem can be solved as a deterministic optimal control problem. Many tools such as dynamic programming and Pontryagin’s minimum principle can be used to solve it.

However, in practice we usually do not know $\bar{w}$ in advance. In our problem, $w_{k}$ is unknown before step $k$. If we do not know the future values of $w$, we can no longer simply apply the deterministic optimal control. Instead, we can consider this sequence $\bar{w}$ as a stochastic signal. It can be represented by a random matrix of dimension $l\times N$, or equivalently, a random vector of dimension $lN$.

The control objective is to find a feedback policy $u=\pi(\cdot)$ that uses available information to minimize the cost, or more rigorously, a certain expectation of the cost. In the context of optimal control in this paper, better control performance means a smaller cost expectation.

To handle the future uncertainties of the disturbance sequence $\bar{w}$, it is common to use a predictor to forecast it. We consider the general case where the forecast result is a discrete or continuous probability distribution of $\bar{w}$. If one specific disturbance sequence is forecasted instead of a probability distribution, we may consider it as a distribution with a one or near-one probability at this specific sequence, and zero or near-zero probability at all other sequences. We call this probability distribution our belief.

We use $\bar{W}_{b}$ to denote this probability distribution of $\bar{w}$. $\bar{W}_{b}\in\bm{\bar{W}}$, where $\bm{\bar{W}}$ is the set of all possible probability distributions of $\bar{w}$. We write $\bar{w}\sim\bar{W}_{b}$, which means $\bar{w}$ follows the distribution $\bar{W}_{b}$. When there is no ambiguity, we do not distinguish the disturbance sequence probability distribution $\bar{W}_{b}$ and its data representation, which may be a high-dimensional vector to represent a probability mass function, or a vector of parameters for a probability density function.

To obtain a belief $\bar{W}_{b}$, we need to observe the environment for necessary information. We assume that $o\in\mathbb{O}$ is the observation from the environment. The observation $o$ may be in the form of sensor readings, images, videos, or data received via communication, and their histories. We can define the concept of predictors as follows.

With the belief generated by the predictor, we can compute the optimal control. As we believe our belief, the expected cost to be minimized can be defined as

where our belief tells us that $\bar{w}\sim\bar{W}_{b}$ and $\bar{W}_{b}=\mathcal{P}(o)$. We use $\pi_{\bar{W}_{b}}(\cdot)$ to denote the optimal policy that minimizes the above expected cost when $\bar{w}\sim\bar{W}_{b}$. This optimal control problem is well-defined, though computing the exact optimal policy $\pi_{\bar{W}_{b}}(\cdot)$ may be challenging if $\bar{W}_{b}$ is complicated. In practice, an approximated solution is usually applied [28].

The predictive optimal control structure shown in Fig. 2 has been used in many control applications: first using a predictor to process the observation $o$ to obtain a belief $\bar{W}_{b}$, then computing (or approximately computing) the corresponding optimal control policy $u=\pi_{\bar{W}_{b}}(\cdot)$, finally applying the control to the dynamic system in (1). The conventional predictive optimal control formulation in practice just stops here, and it cannot provide further insights about the “prediction + optimal control” scheme. It assumes that $\bar{W}_{b}$ is correct, so we use it in our optimal control design. However, since our belief $\bar{W}_{b}$ is estimated, in most cases, it is actually different from the truth.

In real-world applications, usually we know that our belief is imperfect. This brings up a lot of interesting questions. In the rest of this paper, we will focus on the case where we know our belief is not perfect, and we would like to find out how it impacts the control performance, and how we can improve our it.

The true probability distribution of the future disturbance sequence is denoted by $\bar{W}_{t}$. We will define this true probability distribution in Section 3.2. For now, let us assume there exists a well-defined one.

If we are not sure whether our belief is true, we have a problem immediately: since we do not know the true distribution $\bar{W}_{t}$, we cannot evaluate the cost expectation in (3) in the sense that $\bar{w}\sim\bar{W}_{t}$. Based on our imperfect belief $\bar{W}_{b}$, we can only obtain an optimized policy with respect to this imperfect belief. As $\bar{W}_{b}$ is only a subjective probability distribution which is not directly associated with any actual random vectors defined in our formulation so far, we use the notation $\mathop{\hat{\mathbb{E}}}_{\bar{w}\sim\bar{W}_{b}}*(\bar{w})$ to denote the believed expectation of a function $*({\bar{w}})$ when the random vector $\bar{w}$ follows the believed distribution $\bar{W}_{b}$. We will stop using the ambiguous expectation notation in (3) for subject beliefs. Instead, the believed expected cost to be minimized is now defined as

Under this notation, the true expectation can be re-written as

Our optimal policy $\pi_{\bar{W}_{b}}(\cdot)$ obtained using the imperfect belief $\bar{W}_{b}$ minimizes (4), not (5). It initially seems that there is not much we can do if our best estimation is $\bar{W}_{b}$. However, a complete analysis framework will give us insights about the relationship between the predictor and the control performance, thus guide our predictor design.

We assume that the environment that generates the disturbance $w$ is a dynamic system which can be described by the following equations,

where $z_{k}$ is the state of the environment system, $r_{0}$, $r_{1}$, …, $r_{N-1}$ are independent and identically distributed (i.i.d.) random disturbance, $o_{k}\in\mathbb{O}$ is the measured output, which is also called the observation of the environment, and $w_{k}$ is the output of the environment system that impacts our system to be controlled. Since we assume that the value of $w_{k}$ is measured, $o_{k}$ contains all the information of $w_{k}$. From the point of view of the system to be controlled, $w_{k}$ is the generalized disturbance. Since the environment is usually very complex, such as the human beings and the weather system, the dimensions of $z$ and $r$ can be very high and the exact formulas of (6) may never be known.

Let us define the hidden prediction state

as shown in Fig. 3. According to the environment model in (6), the generalized disturbance sequence $\bar{w}$ can be completely determined by the hidden prediction state, which is a random vector, $s\sim S$. Therefore, we write $\bar{w}(s)$ as $\bar{w}$ is uniquely determined when $s$ is given. The observations $o$ can also be completely determined by $s$. Therefore, we write

and use the notations

for short.

In general, $s$ cannot be directly observed at step 0 due to two reasons. First, the environment state $z_{0}$ and environment disturbance $r_{0}$ may not be estimated by only measuring $o_{0}$. Second, the i.i.d. random disturbances $r_{1}$, $r_{2}$, …, $r_{N-1}$ at future steps cannot be known at step 0.

Since there may be multiple ways building the environment model, the selection of the hidden prediction state $s$ is not unique. In predictive optimal control applications, it is not necessary to formulate the environment and define the hidden prediction state at all. However, the environment model in (6) and the definition of the hidden prediction state $s$ in (7) are the cores of the analysis framework, and they can help us in analyzing the problems when the predictions are not perfect.

In the previous section, we assume that there is a true probability distribution $\bar{W}_{t}$. Now let us define it with the environment model.

We consider the belief obtained at step 0. With the environment in (6), our belief $\bar{W}_{b}$ is determined by applying the predictor $\mathcal{P}$ on the observation $o(s)$, therefore we can write

When there is no ambiguity, we may also write $\mathcal{P}(s)$ instead of $\mathcal{P}(o(s))$ for conciseness. We investigate multiple potential ways to define the true probability distributions.

In the predictive optimal control implementation under the structure in Fig. (2), the prediction runs recurrently at every control step. Depending on whether new observations are used after the initial step and how long the prediction horizon is, we consider three typical types of recurrent predictions.

An accurate predictor means that when the predictor says the probability distribution of $\bar{w}$ is $\bar{W}_{b}$, the distribution is indeed $\bar{W}_{b}$. However, just being accurate is not good enough for predictors. To show this, we will define it rigorously first.

This definition means that if we collect all realized disturbance sequence data for any fixed belief from an accurate predictor, the collected data distribution will match this belief. Simply being accurate does not mean a good predictor. For example, a blind predictor is a predictor that generates the same belief for all observations. An accurate blind predictor does not use any information from the observation $o$, yet it offers accurate beliefs. It is accurate, but not very informative.

The goal of prediction is to obtain the true distribution $\bar{W}_{to}(o)$ for every observation $o$. To show this, we will prove that if our belief is the same as $\bar{W}_{to}(o)$, we obtain the optimal control performance, as long as we use no more information about $s$ than $o(s)$.

Given a predictor $\mathcal{P}$, the predictive optimal control cost expectation is $\mathbb{E}\,J^{\pi_{\mathcal{P}(S)}}_{\bar{w}(S)}$. By the law of total expectation and the definition of $W_{to}(o(s))$,

By definition, $\pi_{\bar{W}_{to}(o)}$ is the optimal strategy that minimizes $\mathop{\hat{\mathbb{E}}}_{\bar{w}\sim\bar{W}_{to}(o)}J^{(\cdot)}_{\bar{w}}$ for every $o$. Since we use no more information about $s$ than $o(s)$, given any $o$, this cost is the best that we can achieve. Therefore, given any $o$, $\bar{W}_{to}(o)$ minimizes the predictive optimal control actual cost $\mathop{\hat{\mathbb{E}}}_{\bar{w}\sim\bar{W}_{to}(o)}J^{\pi_{(\cdot)}}_{\bar{w}}$. So it minimizes the predictive optimal control cost expectation.

The goal of our predictor result $\mathcal{P}(o)$ is $\bar{W}_{to}(o)$. However, the path to achieve this goal is full of challenges. First, in practice, we may never be able to directly compare the predicted belief $\mathcal{P}(o)$ with our target $\bar{W}_{to}(o)$ as $\bar{W}_{to}(o)$ is unknown. Second, in general, there is no guarantee that predictors generating beliefs closers to $\bar{W}_{to}(o)$ lead to better control performance. Even locally around $W_{to}(o)$, as long as $\mathcal{P}(o)\neq\bar{W}_{to}(o)$, we do not have such guarantees (details will be discussed in Section 5 and examples are provided in Section 6). Nevertheless, this ultimate goal still provides us an incentive to keep optimizing the predictor towards $\bar{W}_{to}(o)$.

In most predictive optimal control applications, it is not possible to obtain enough data to estimate $\bar{W}_{to}(o)$ for every $o$. In practice, the data samples are in the form of $\{o,\bar{w}\}$ pairs. For a few specific $o^{i}$, we may have many $\{o^{i},\bar{w}\}$ pair samples such that $\bar{W}_{to}(o^{i})$ can be directly estimated. However, for most $o^{i}$, we usually do not have enough $\{o^{i},\bar{w}\}$ samples to estimate $\bar{W}_{to}(o^{i})$. For example, in human-driven vehicle speed prediction, where the to-be-predicted $w$ is the vehicle speed, and the observation $o$ is the driving scenario data including the vehicle status, driver status, road conditions, and traffic conditions, etc. We can collect many samples in the form of $\{o,\bar{w}\}$ pairs under many driving scenarios. But for a specific driving scenario, it is very difficult to have repeated data showing how a driver behaves differently each time, as the scenario keeps changing. In practical situations, we usually make some assumptions on the predictor and parameterize the predictor so that it can be trained using $\{o,\bar{w}\}$ pairs. We also need to evaluate predictor performance using many $\{o,\bar{w}\}$ pairs, while not assuming we have repeated data for every specific $o^{i}$ to learn $\bar{W}_{to}(o^{i})$.

With one $\{o,\bar{w}\}$ pair, given a predictor $\mathcal{P}$, we can compute a belief $\bar{W}_{b}=\mathcal{P}(o)$. Using a beleif $\bar{W}_{b}$ and the corresponding realized disturbance sequence $\bar{w}$, we can define some one-time prediction performance measures. With multiple $\{o,\bar{w}\}$ pairs, the predictor $\mathcal{P}$’s performance can be evaluated by aggregating these one-time prediction performance measures.

if we choose the expected MSE as the one-time prediction performance measure, then,

In this case, $\bar{W}_{b}=\bar{W}_{to}$ may not even locally minimize $E_{M}(\bar{W}_{to},\cdot)$. This means that when using MSE, even with a large amount of data, we will miss the preditor’s ultimate goal $\bar{W}_{to}$. When there is no constraint, the best $\bar{W}_{b}$ may be a deterministic prediction, which is inferior to the best stochastic predictor in terms of the control performance. With the proposed framework, examples can be easily constructed to show that the MSE-based measure is neither a best-P-lowest-C measure nor a better-P-lower-C measure (see Section 6).

If the regret-based predictor performance measure is used, then

The second term $\mathop{\hat{\mathbb{E}}}_{\bar{w}\sim\bar{W}_{to}}J^{\pi_{\bar{w}}}_{\bar{w}}$ is the posteriori optimal cost, which is a constant given $\bar{W}_{to}$. The first term is the same as the expected cost. The regret is essentially the control cost with a constant offset. $\bar{W}_{to}$ globally minimizes $E_{R}(\bar{W}_{to}(o),\cdot)$. Furthermore, decreasing $E_{R}(\bar{W}_{to}(o),\bar{W}_{b})$ will lead to a decrease of the expected cost. Evaluating the predictors using regret-based error measures is essentially evaluating the control performance after connecting the prediction and the optimal control process. Actually, we can just compute the first term $\mathop{\hat{\mathbb{E}}}_{\bar{w}\sim\bar{W}_{to}(o)}J^{\pi_{\bar{W}_{b}}}_{\bar{w}}$, which is the control cost under the predictor, to evaluate this predictor. The regret is both a best-P-lowest-C and a better-P-lower-C measure.

If we use the probability-based measure log-likelihood $P$, in the discrete case,

This expectation is related to two probability distributions: the subjective belief and the observable true probability distribution (it is different from the entropy $-\sum_{\bar{w}}\hat{\mathbb{P}}_{\bar{W}_{b}}(\bar{w})\log\hat{\mathbb{P}}_{\bar{W}_{b}}(\bar{w})$, which describes the property of one probability distribution). The best predictor is $\hat{\mathbb{P}}_{\bar{W}_{b}}(\bar{w})=\hat{\mathbb{P}}_{\bar{W}_{to}}(\bar{w})$ for all $\bar{w}$ due to convexity, which means this measure is best-P-lowest-C. However, there is no guarantee to make it better-P-lower-C.

A summary of the three predictor measures is provided in Table 1. Neither the MSE nor the log-likelihood measure have the Better-P-lower-C property. For a general predictive control problem, , there is no guarantee that predictors with better MSE or log-likelihood lead to better control performance. It implies that the predictor design cannot be simply decoupled from the downstream optimal control problem, and the predictor needs to be evaluated along with the control system performance, e.g., using the control cost or the regret as the predictor performance measure.

We provide a numerical example where the predictor measure gets better while the control cost gets worse in a Type I problem, and illustrate the differences of the three predictor measures shown in Table 1. Consider the simple linear system,

where $-1\leq u\leq 1$, $x_{0}=w_{0}=0$, with a quadratic cost function $J=x_{2}^{2}$. There is no observation information other than $w$. The input $u_{1}$ will be determined after $w_{1}$ is measured. So the key of this predictive optimal control is to forecast the value of $w_{1}$ and determine $u_{0}$ accordingly.

Assume that $w_{1}\in\{-3,2\}$. Therefore, the observable true probability distribution of $w_{1}$ is in the form of

In addition, we assume that $0\leq p\leq\frac{2}{3}$. In this example, the hidden prediction state $s$ can be considered as $w_{1}$. If we know the value of $p$ and can use it to design the optimal control accordingly, the ideal optimal policy is

and the ideal optimal cost expectation is,

A predictor gives a prediction of $p$ as $p_{b}$ and $0\leq p_{b}\leq\frac{2}{3}$. $p_{b}$ generates a belief of $w_{1}$. The predictive optimal solution is in the same form as (36) while replacing $p$ with $p_{b}$. With this belief, the predictive optimal control cost expectation is

Obviously, $p_{b}=p$ minimizes this cost expectation.

When using the MSE, the regret, and the log-likelihood measure, the predictor performance measures are

respectively. In the regret and the log-likelihood, $p_{b}=p$ minimizes the measures, as they are best-P-lowest-C measures. Furthermore, for the regret, $E_{R}(\bar{W}_{to},\bar{w}_{b})$ is just the cost expectation $\mathbb{E}\,J^{\pi_{\mathcal{P}(o(S))}}_{\bar{w}(S)}$ with a constant offset. However, in MSE, depending on the sign of $1-2p$, $p_{b}$ takes its minimal MSE value at $0$ or $\frac{2}{3}$, which is different from its optimal result $p$ in terms of the optimal control cost.

We use $p=0.3$ to illustrates the better-P-lower-C properties of the measures. The trend of the predictor measures and the cost expectation is shown in Fig. 6 as the belief $p_{b}$ changes. In the MSE and the log-likelihood case, it is possible to improve the predictor measure while making the optimal control cost worse.

We use a hybrid electric vehicle example in the form of a Type III problem to demonstrate the relationship between predictors and control performance in the real world. Given a pre-defined driving cycle as a time-velocity table, optimal control can be used to determine the most efficient energy management strategy for a hybrid electric vehicle. With uncertain future driving cycles, predictive optimal control can be used and the future vehicle velocities are forecasted. We use real-world driving data from the Next Generation Simulation (NGSIM) [29]. We consider the energy management strategy for a simple hybrid electric vehicle powertrain shown in [30]. The hybrid electric vehicle powertrain model is as follows,

where $\eta_{b}=\eta_{b}(x_{SOC,k},\omega_{k},T_{m,k})$, $\eta_{m}=\eta_{m}(\omega_{k},T_{m,k})$, $x_{SOC}$ is the battery state of charge (SOC), $K$ is a constant, $\eta_{b}$ is a variable representing the battery or the inverse of the battery efficiency, depending on the sign of the motor power, $\eta_{m}$ represents the motor efficiency or the inverse of the motor efficiency, $\omega$ is the motor speed, $T_{m}$ is the motor torque. We assume the following static relationship is known,

where $T_{d}$ is the total torque demand to the powertrain, $v$ is the vehicle velocity, $a$ is the vehicle acceleration. The control input is the motor torque $T_{m}$. Once $T_{m}$ is determined, the engine torque $T_{e}$ is determined as

The cost function is defined as

where $\alpha_{1}$, $\alpha_{2}$ and $\alpha_{3}$ are weights, $J_{FC}$ is the fuel consumption determined by $\omega$ and $T_{e}$, and $x_{target}$ is the target battery SOC to be maintained. This cost function considers the electric energy consumed, the fuel consumed, and the battery SOC variation from the target value during the $N$ steps.

In this problem, $w=[v,a]^{T}$ (or equivalently, $w=[\omega,T_{d}]^{T}$) is the generalized disturbance. Observation $o$ is just $w$ and its history. A predictor tries to forecast future $w$, that is, the future velocity and acceleration. The future velocity and acceleration is dependent on various factors including the driving style, the road and traffic condition. In the proposed analysis framework, these factors are considered in the environment dynamics equations in (6). The hidden prediction state $s$ associated with this environment dynamic system may be a high-dimensional vector. But the exact formulas associated with the environment and the hidden prediction state are not needed for applying predictive optimal control.

Four types of deterministic predictors and two types of stochastic predictors are designed.

(D1) Constant velocity predictor.

(D2) Linear decay acceleration predictor.

where $\gamma$ is a constant parameter.

(D3) Exponential decay acceleration predictor.