Probabilistic Charging Power Forecast of EVCS: Reinforcement Learning Assisted Deep Learning Approach

The target of our studied probabilistic forecast problem is modelled by following formulations.

where $\widetilde{F}_{t+1}$ is the probabilistic forecast of EVCS charging power at time $t+1$. $\hat{E}_{t+1}$ is the expectation of the forecast values, which is obtained by the point forecast function $f(\cdot)$, as shown in (3) [19]. $\epsilon$ represents the noise, which stands for the uncertainty between the forecast and real values. In this paper, it is assumed that the $\epsilon$ is Gaussian distributed with variance $\delta_{t+1}$, which is obtained by the variance forecast function $g(\cdot)$, presented in (4). It has been proved that the time series models based on the Gaussian distribution assumption can be applied to obtain a satisfactory performance [21]. Note that $f(\cdot)$ and $g(\cdot)$ represent the LSTM and our proposed AePPO, respectively. In addition, $X_{t}=\{x_{t-T},...,x_{t-1},x_{t}\}$ is the vector of input features of the two functions. It stands for the features of $T$ timestamps before $t+1$, whereas $x_{t}\sim\mathbf{N}^{N_{f}}$ ($\mathbf{N}^{N_{f}}$ denotes the $N_{f}$ dimensional integer space).

Generally, $f(\cdot)$ provides a point forecast which represents the mean value of our target [22]. However, the obtained results might be unreliable since its inherent parameters are learned by the stochastic gradient descent optimization method such as Adam [18] and its input data $X_{t}$ is related to the stochastic situation, i.e., the charging behavior of EV users. Therefore, it is necessary to quantify the uncertainty of $f(\cdot)$ and $X_{t}$, and the introduction of the noise $\epsilon$ is one of the most effective approaches [21, 23].

Usually, the $\delta_{t+1}$ is forecasted through statistical indicators, which requires a massive repeated running of $f(X_{t})$ and thus leads to a huge computation complexity [19]. Consequently, it is expected to design a function $g(\cdot)$ to forecast the variance $\delta_{t+1}$ without the repeated running. The $g(\cdot)$ should have the ability to capture the uncertainty of $f(\cdot)$ and $X_{t}$, which means the input of $g(\cdot)$ has to contain the characteristics of both point forecast model and data. Compared with other deep learning algorithms, such as ANN and conventional neural network, the LSTM owns a unique structural parameter, i.e., the cell state, which is the core parameter of LSTM and is also determined by the $X_{t}$. Therefore, the uncertainties coming from both $f(\cdot)$ and $X_{t}$ would make changes to the value of cell state, which could be captured by the $g(\cdot)$ to forecast $\delta_{t+1}$. Besides, since the value of cell state is affected by its previous value, the change is modelled as a MDP in this paper. Therefore, the cell state is denoted as the state of the MDP, and the $\delta_{t+1}$ is defined as the action. In this way, a reinforcement learning model, AePPO, is developed as $g(\cdot)$ to observe the cell state and generate the action, i.e., the $\delta_{t+1}$. Generally, when the LSTM is applied to produce $\hat{E}_{t+1}$, the cell state would be produced, and the AePPO would capture it to determine $\delta_{t+1}$. Then, the probabilistic forecast can be constructed by $\hat{E}_{t+1}$ and $\delta_{t+1}$.

As discussed above, the LSTM and AePPO are trained and utilized to obtain the mean and variance values for obtaining the predicted probabilistic distribution of EVCS charging power. In order to implement this procedure, we propose a probabilistic forecast framework, as shown in Fig. 1. It contains three parts, preprocessing, training, and utilization. They are presented by the three sub-figures in Fig. 1, which are denoted as (a), (b) and (c), respectively.

As illustrated in Fig. 1(a), an EVCS has numerous chargers which record the information about charging power in the period between the arrival and departure of an EV user [17]. Since all chargers may work simultaneously, the charging information of EVCS is obtained by recordings of chargers. In this paper, such recording is termed as the charging session. It consists of the period during the process of EV charging, i.e., the arrival and departure time of the EV, and the related data about the energy information, for instance, the demand and remaining energy of the corresponding EV. Note that one charging session only contains partial charging information of EVCS and the period between different charging sessions may overlap. The information in charging session during the overlap period should be gathered to derive the data of EVCS for LSTM training. In this case, the charging session data should be pre-processed by aggregating the information and transforming them into the time-series format with different features, such as the charging power, utility time and demand satisfaction rate.

During the training process shown in Fig. 1(b), the LSTM and AePPO are separately trained. At first, the EVCS data is used to train LSTM. Then, for the well-trained LSTM, its variation of cell state is modeled as a MDP. By doing so, AePPO could produce the action and interact with the LSTM according to the state. Afterwards, the reward is calculated based on the state and action, which is used for the training of AePPO.

Finally, the last part of this framework is the utilization, as shown in Fig. 1(c). The LSTM and AePPO models are utilized here. That is, the LSTM provides the mean value of probabilistic forecast distribution $\hat{E}_{t+1}$, while AePPO is used to determine the variation $\delta_{t+1}$ based on the cell state of LSTM. In this way, the predicted probabilistic distribution $\widetilde{F}_{t+1}$ could be obtained by Eq. (1).

In this paper, in order to obtain the mean value of EVCS charging power for the forecast of probabilistic distribution, the LSTM is applied. Compared with other deep learning algorithms, such as the fully connected network, convolutional neural network and vanilla recurrent neural network, LSTM is more capable of learning the long-term dependencies inherent to the time-series data and would not suffer from the vanishing gradients [23]. Besides, Overall, the LSTM is formulated as follows.

where $y_{t+1}$ represents the output of LSTM at time $t+1$, which stands for the predicted mean value of EVCS charging power. The LSTM model A takes $X_{t}$, $y_{t}$ and $c_{t}$ as inputs, which denote the input data, the output and the cell state of LSTM at time $t$, respectively.

For clarity, the formulations of the LSTM is referred as follows.

where $\left[W_{i},W_{f},W_{o},W_{c}\right]$ are the learnable weight metrics and $\left[b_{i},b_{f},b_{o},b_{c}\right]$ are the learnable bias metric. $\sigma$, tanh are the sigmoid and hyperbolic tangent activation functions. Their output range is $(0,1)$. $\odot$ indicates the elementwise multiplication. $i_{t+1},f_{t+1},o_{t+1}$ stand for the input, forget and output gate, respectively. ${\tilde{c}}_{t+1}$ represents the candidate information of the inputs in $c_{t+1}$, i.e., the cell state.

It can be seen from Eqs. (14) and (15) that the value of the cell state highly influences $y_{t+1}$. Besides, these equations also show that the cell state is determined by the learnable parameters and the input data $X_{t}$, and they are affected by the uncertainty from the model and user’s behaviour, respectively. Wherein, the uncertainty of the LSTM model comes from the learnable parameters, which are determined by the stochastic gradient descent optimization method. In addition, since $X_{t}$ is aggregated by the charging session data, it is uncertain as well because of correlations with the behaviour of users.

As described above, the input of LSTM $X_{t}$ is the time-series format, which reflects the behavior of the users. Besides, because of the recurrent structure of the LSTM, the cell state is influenced by its previous value. Therefore, the variation of the LSTM cell state is caused by the uncertainty that comes from both the model and the data. In this case, the variation of the cell state should be modelled to extract the variance of the probabilistic forecast distribution from the LSTM cell state. Since the cell state is determined by its previous state, its variation can be modelled as a MDP. Normally, the MDP is represented by a tuple $\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}\rangle$. $\mathcal{S}$ is the state space that stands for all possible states of the environment. $\mathcal{A}$ represents the action space, i.e., the agent interacts with the environment to produce action $a\in\mathcal{A}$ for the guidance of state transition. $\mathcal{P}=\{p(s_{t+1}|s_{t},a_{t})\}$ stands for the set of transition probability, and $\mathcal{R}=r(s_{t},a_{t}),\mathcal{R}\in\mathbb{R};\mathcal{S\times A}\rightarrow\mathbb{R}$ depends on the state and action and is termed as the reward. In our problem, the MDP can be formulated as follows:

1) Environment: The environment produces the state $s_{t}$ and computes the reward $r_{t}$ based on the action $a_{t}$ of the agent. In this paper, the variance of the probabilistic forecast distribution is obtained from the variation of the cell state, which is the most representative parameter of the LSTM. The definition requires a premise, namely, the LSTM model is well-trained and its learnable parameters are fixed. The promise ensures the decisive position of the cell state in the model, therefore, a well-trained LSTM is defined as the environment of MDP.

2) Agent: Here, the agent stands for a policy to obtain the action from observing the state. Besides, the agent could imply the self-learning from its experience to gradually improve the policy.

3) State: The definition of the state $s_{t}$ at time $t$ should follow two criteria. First, the state should be the most representative parameter of the environment. Second, the state should only be related to the previous state and not affected by the next state, namely, the non-aftereffect property. According to the calculation process of LSTM shown in Eqs. (10)$\sim$(15), the most suitable parameter is the cell state, which not only determines the output of LSTM but also be obtained by the cell state of the previous time. Therefore, $s_{t}$ is set as $c_{t}$ and the state space equals to all the possible value of $c_{t}$.

4) Action: As the uncertainty of the model and input data are contained in $s_{t}$, it is expected that the agent can capture this uncertainty to produce the variance of the forecast distribution, i.e., the action $a_{t}=\delta_{t}$.

5) Reward: The reward $r(s_{t},a_{t})$ indicates the evaluation index of $a_{t}$ on the basis of the $s_{t}$. It gives feedback to the agent about the performance of its action, which could be used to update the agent. Since our target is to obtain the probabilistic distribution of future charging power, which is constructed by the output of the environment, i.e., LSTM and the agent’s action. In our case, the reward is designed to evaluate the performance of the predicted distribution.

To quantify it, the continuous ranked probability score (CRPS) is introduced to measure of performance for probabilistic forecasts, i.e., the accuracy of the predicted probabilistic distribution [24]. The details of CRPS calculation are given in the Appendix A.

Note that the lower CRPS represents the more accurate probabilistic forecast. Therefore, the reward is defined as $r=-\zeta\times\text{CRPS}$, which should be maximized by the agent. $\zeta$ stands for the shrinkage coefficient, which is set as 0.75.

Based on the MDP, a recent reinforcement learning algorithm, called PPO, is introduced to train the agent for generating the optimal policy $\pi(a_{k}|s_{k})$. The PPO contains two types of deep neural networks, which are termed as actor and critic. The actor works as the agent, and it produces the action by a policy $\pi$ with parameter $\theta^{\pi}$. On the other hand, the critic is used to evaluate the performance of actor, parametrized by $\theta^{Q}$. The critic approximates the value function $V^{\pi_{{k}}}(s_{k})$, which evaluates the value of the state $s_{k}$, i.e., it represents the expectation of the reward from $k$ to the end of the MDP, which can be formulated as follows.

where $U_{k}$ is the return function, which is defined as the discounted cumulative reward from state $s_{k}$.

where $\gamma\in\left[0,1\right]$ stands for the discount factor, $r_{t}$ denotes the value of reward and $T$ is the finite time horizontal of the MDP.

The main target of the PPO is to train the actor and critic, by learning from the experience tuples $\langle s_{k},a_{k},r_{k},s_{k+1}\rangle$. It is obtained by the interaction between the actor and the environment. The actor generates a Normal distribution $\mathcal{N}(a_{k,\text{mean}},a_{k,\text{var}})$ after observing $s_{k}$. That is, the action $a_{k}$ is sampled from this distribution, i.e., $a_{k}\sim\mathcal{N}(a_{k,\text{mean}},a_{k,\text{var}})$. Therefore, this introduces randomness in the produced action and thus create more diverse actions when observing the same state. In this way, it leads to the diversity of rewards since they are related to the action. This mechanism is termed as the exploration of PPO, which is used to enrich the experience tuple in order to prevent the actor from getting trapped in local optima.

Then, based on the prediction of the mean value $y_{k}$ obtained by LSTM, the predicted probabilistic distribution $\mathcal{N}(y_{k},a_{k})$ is constructed. Afterwards, $r_{k}$ could be calculated by the reward function to evaluate $\mathcal{N}(y_{k},a_{k})$, and the state at the next time $s_{k+1}$ is also restored. With the experience tuple, the parameters of both networks are updated by following equations.

where $\eta_{\pi}$ and $\eta_{Q}$ denote the learning rates of actor and critic, respectively. $\mathcal{L}^{\text{CLIP}}$ and $\mathcal{L}^{\text{Q}}$ stand for the loss functions of the two networks, which are given by

where $\pi_{k-1}(a\mid s)$ denotes the policy of $(k-1)$th iteration, and $\gamma$ is the discount rate. $\text{clip}(t,t_{\text{min}},t_{\text{max}})$ is the clip function. It returns $t_{\text{max}}$ if $t>t_{\text{max}}$, and $t_{\text{min}}$ if $t<t_{\text{min}}$. $A^{\pi_{{k}}}_{s,a}$ is the advantage of action $a$ under state $s$, which is represented by the difference between $a$ and the averaged performance of the actor, formulated as follows.

The Q-function $Q(s_{k},a_{k})$ evaluates the value of executing action $a_{k}$ at state $s_{k}$, i.e., it represents the expectation of the reward from $s_{k}$ to $s_{T}$ after selecting the action $a_{k}$. The formulation of the $Q$ is shown below.

Then, the $Q(s_{k},a_{k})$ is calculated by the following equation in the PPO algorithm.

In this part, we conduct a case study to verify the effectiveness of our proposed algorithm, i.e., LSTM-AePPO. The training data of the algorithm is collected from ACN dataset [26], which is an open dataset for EVCS charging researches. The charging sessions in ACN dataset are recorded from two EVCSs, one at Caltech, Pasadena, including 54 chargers and the other located in Jet Propulsion Laboratory (JPL), Lacanada, USA, containing 50 chargers. In our experiments, the charging session from 1st June 2018 to 1st June 2020 are applied to train the LSTM-AePPO, in which 25916 and 22128 charging sessions are included in the case of Caltech and JPL, respectively.

In order to demonstrate the effectiveness of LSTM-AePPO, numerous algorithms, i.e., support vector quantile regression (SVQR) [27], linear quantile regression (QR) [28], and gradient boosting quantile regression (GBQR) [29] are introduced for comparisons. Considering the seasonal variations of charging behavior, the performance and metric comparisons are conducted on the four seasons, respectively. Moreover, for further verifying the outperformance of our proposed AePPO, the traditional PPO is introduced to generate the $\delta_{k+1}$ from the cell state for comparisons, which is termed as LSTM-PPO.

Then, CRPS, Winkler [23], and Pinball [30] are introduced as evaluation metrics. The three metrics focus on the accuracy, variation, and reliability of the forecast distribution, respectively. Their details are presented in Appendix C.

In this paper, the features of former $N_{h}$ hours $[x_{i-1},...,x_{i-N_{h}}]$ are used in the probabilistic forecasting of EVCS charging power at hour $i$. The value of $N_{h}$ is set as 12, and other hyperparameters of LSTM-AePPO are given in Table I. Note that the experiments are conducted on a computer with 8GB RAM, Intel i5-8265U CPU, and implemented in Python.

The probabilistic forecast of EVCS charging power obtained by LSTM-AePPO are shown in Fig. 5 and Fig. 6, which illustrate the results on Caltech and JPL cases among 3 test days (72 hours). The subfigures (a), (b), (c) and (d) represent the forecast of Spring, Summer, Autumn, and Winter, respectively. Specifically, $p\%$ prediction interval (PI) is used to denote the interval between the lower and upper quantiles at probability $p$, and the darker color denotes the PI with a smaller $p$. Consequently, the 30% PI is shown with the darkest color while the 90% PI corresponds to the lightest. Note that the real charging power is represented by the red line in these figures.

One of the biggest challenges of the charging power forecasting is to predict the peak, which varies according to the seasons. For instance, in the Spring and Autumn case of Caltech, it could be learned that the peaks emerge in around the 18th and 42nd hours, as shown in Figs. 5(a) and (c). However, as shown in Figs. 5(b) and (d), the peaks during the Summer and Winter appear 3 times, which is more frequent. Besides, as shown in Fig. 6, the peaks are more irregular in the JPL case, which appear 2 times in the Spring and Autumn cases, 1 time in the Summer case and 3 times in the Winter case. Although the peak appears irregularly, LSTM-AePPO achieves a good result. The fluctuation of the probabilistic forecast distribution can capture the EVCS charging power. As shown in Figs. 5 and 6, most of the values in the red lines are covered by the 90% PI, which means that the probabilistic forecast distribution could represent the variation of real charging power.

To verify the effectiveness of LSTM-AePPO, the Winkler, Pinball and CRPS are introduced as metrics for comparison. Since the values of Winkler and Pinball are related to PI, the effectiveness of our proposed algorithm is demonstrated by comparing with other algorithms under $30\%$, $60\%$ and $90\%$ PIs, as shown in Table II.

In this table, the best metric value of each case is marked in bold. It could be seen that the LSTM-AePPO outperforms other algorithms on Winkler and Pinball in most cases. For example, in the Spring case of Caltech, the Winkler and Pinball values of LSTM-AePPO at 30%, 60% and 90% PIs are (30.099, 15.158), (21.036, 13.311) and (23.966, 14.057), which are lower than the Winkler values of other algorithms. Besides, in the Summer and Autumn cases of Caltech, the LSTM-AePPO always performs the best at 60% PI. Moreover, when focusing on the Winter case of Caltech, the QR, QRSVM, GBQR and LSTM-PPO also perform worse than LSTM-AePPO since their Winkle and Pinball values are larger. Generally, the forecast distribution obtained by LSTM-AePPO have a better variation and reliability at Caltech case. In addition, the effectiveness is also verified comprehensively in the JPL case. As shown in the Table II, the Winkler and Pinball numbers obtained by LSTM-AePPO in the JPL case are bold, which means QR, QRSVM, GBQR and LSTM-PPO are all surpassed from both the variation and reliability of the forecast distribution.

Note that the LSTM-AePPO does not perform the best in the following two cases, i.e., the Winkler of Caltech Autumn case at 90% PI (21.510 for LSTM-AePPO but 18.342 for LSTM-PPO) and the Pinball of Caltech Summer case at 30% PI (23.475 for LSTM-AePPO but 21.962 for QRSVM). The cause of this phenomenon may be the correlation between Winkler and Pinball to the PI. Therefore, as a metric that is independent of PI, the CRPS is introduced to evaluate the accuracy of the whole forecast probabilistic distribution, and the lower CRPS value indicates a more accurate forecast probabilistic distribution. The comparisons of CRPS under Caltech and JPL cases are illustrated in Fig. 7 and Fig. 8. It can be seen from the two figures that our proposed LSTM-AePPO achieves the lowest CRPS values in all the cases, which demonstrates the accuracy of LSTM-AePPO could surpass other comparison algorithms. Based on the analysis of the Winkler, Pinball and CRPS values under different cases, it could be concluded that the LSTM-AePPO is more effective than QR, GBQR, QRSVM and LSTM-PPO.

To illustrate the effectiveness of the proposed adaptive exploration mechanism in LSTM-AePPO, the reward curves of PPO and AePPO during the 10000 training iterations are compared in the Caltech and JPL cases, as shown in Fig. 9 and Fig. 10. In Fig. 9, the reward of PPO increases during the first 3000 iterations and stays steady. Similarly, as shown in Fig. 10, the reward of PPO continuously raise from -20 to -7.5 in the first 1000 iterations then maintains at a relatively stable level.

Because of applying the adaptive exploration mechanism, the AePPO would focus on exploration in the early stage, which enriches the experience while causing the backwardness of the reward value. For instance, in the case of Caltech, the reward of PPO increases from around -30 to -23 during the early 3000 iterations, while the reward value of AePPO raises from -29 to -27. The same situation also appears in the case of JPL. The reward of PPO goes up from -25 to -7 in the early training, while the corresponding reward of AePPO hardly increases. However, the exploration of PPO would decrease due to the premature convergence of its actor, thus the diversity of its experience is not plentiful enough, which may cause the PPO falling into the local optima. As a result, the rewards of AePPO increase and surpass PPO after around the 8000th and the 5000th iteration in the case of Caltech and JPL, respectively.

Therefore, it can be seen that the adaptive exploration mechanism strengthen the accumulation of AePPO experiences in the early stage. Then, the AePPO would make full use of these experiences during the training and finally achieves a higher reward than PPO. In addition, the fluctuation of reward curves obtained by LSTM-AePPO are lower in the two cases. This illustrates that the proposed AePPO provides a more stable and superior performance than the original PPO.