Event-based EV Charging Scheduling in A Microgrid of Buildings

We consider a microgrid of buildings as depicted in Fig. 1. In the microgrid, each building is equipped with distributed renewable energy (DRE), hydrogen energy storage (HES) and charging piles. The building should provide charging service and keep load balance. We assume that only when the output of DRE and HES cannot satisfy the EV charging demand and building load, the building will procure power from the grid through microgrid operation controller. The building can also sell power to the grid if the output of DRE is large. In order to reduce the EV charging impact and ensure transmission safety, microgrid operation controller should regulate EV charging behaviors in each building based on the supply and demand status.

In the following, we will first formulate this multi-stage stochastic decision problem as Markov decision process to capture the uncertain renewable energy and EV charging demand in each building. To simplify the discussions, the following assumptions are made:

1) The charge level of each EV is fixed.

2) The distributed renewable energy is free.

3) The electricity price from the grid is deterministic and known a priori.

We consider this EV scheduling problem in a microgrid of buildings over the discretized horizon $t=1,2,...,T$ where $t$ denotes the decision epoch and $\Delta T$ denotes the decision interval. There are $K$ buildings and $N$ EVs in the microgrid. The MDP model of the proposed problem is shown below.

1) System States: The system state at stage $t$ is defined as $S_{t}=[s_{t}^{1},s_{t}^{2},...,s_{t}^{K}]$ where $k=1,2,...,K$ and $s_{t}^{k}$ denotes the state of the $k$th building. For each $s_{t}^{k}$, it is defined as $s_{t}^{k}=[r_{t}^{k},b_{t}^{k},n_{t,m}^{k},n_{t,c}^{k}]$ where $r_{t}^{k}$ denotes the output of DRE in the building, $b_{t}^{k}$ denotes the State of Charge (SOC) of HES, $n_{t,m}^{c}$ denotes the number of EVs which must be charged at stage $t$ and $n_{t,c}^{c}$ denotes the number of EVs which can be charged at stage $t$.

2) Action Space: The control action at stage $t$ is defined as $A_{t}=[a_{t}^{1},a_{t}^{2},...,a_{t}^{K}]$ where $a_{t}^{k}$ denotes the specific action for the $k$th building. Each building should decide whether to provide charge service for the connected EVs at stage $t$. Therefore, there is $a_{t}^{k}=[z_{t,1}^{k},z_{t,2}^{k},...,z_{t,N}^{k}]$ where $z_{t,i}^{k}\in\{0,1\},i=1,2,...,N$. When $z_{t,i}^{k}=1$, it means the $k$th building should provide charge service for the $i$th EV at stage $t$ if it is parked, otherwise $z_{t,i}^{k}=0$.

3) System Dynamics: As the energy status of EV and HES are both time-coupled, their system dynamics should be considered when action $A_{t}$ is decided for the current state $S_{t}$.

For each EV, we use a tuple $(T_{t}^{i},E_{t}^{i},L_{t}^{i})$ to represent its remaining parking time, remaining required charging energy and parking location. $L_{t}^{i}\in\{0,1,2,...,K\}$ and $L_{t}^{i}=0$ if the $i$th EV is on the road. Then, there is

where $P$ is the charge power, $\psi^{c}$ denotes the charge efficiency, $\tau_{t+1}^{i}$ and $\eta_{t+1}^{i}$ are both nonnegative random variables which denote the stochastic characteristic of EV charging demand in the future. As the location transitions for EVs are time-variant, the location transition probability can be established as

where $p_{K1}(t)$ denotes the EV is parked in the $K$the building at stage $t$ and moves to the first building, and so forth.

According to [20], the system dynamics of HES in each building can be depicted as follows

where $\kappa_{t}^{k}$ is the stored hydrogen of HES in the $k$th building at stage $t$, $\kappa_{\text{cap}}$ is the hydrogen storage capacity of HES, $\varphi^{\text{H2P}}$ is the round-trip efficiency from hydrogen to power, $\varphi^{\text{P2H}}$ is the round-trip efficiency from power to hydrogen, $h_{t}^{k}$ is the discharge power of HES if $h_{t}^{k}\geq 0$, otherwise is the charge power of HES. Considering $\kappa_{e,t}^{k}=\kappa_{t}^{k}\sigma_{H_{2}}$ where $\kappa_{e,t}^{k}$ is the stored hydrogen energy and $\sigma_{H_{2}}$ is the lower heating of hydrogen, equation (4) can be rewritten as follows by multiplying both sides with $\sigma_{H_{2}}/\kappa_{e}^{\text{cap}}$

where $\kappa_{e}^{\text{cap}}=\kappa^{\text{cap}}\sigma_{H_{2}}$ denotes the energy capacity of HES, $\eta^{dc}=\varphi^{\text{H2P}}/\sigma_{H_{2}}$ denotes the discharge efficiency of HES and $\eta^{c}=\varphi^{\text{P2H}}\sigma_{H_{2}}$ denotes the charge efficiency of HES.

For each HES in the building, we regulate that HES will discharge when the DRE cannot satisfy the building demand and will charge if DRE is sufficient to meet the building demand, i.e.,

where $l_{t}^{k}$ denotes the net demand in the $k$th building, $p_{t}^{k}=\sum_{i=1}^{N}z_{t,i}^{k}P$ is the total charge power in the $k$th building, $\overline{h_{t,c}^{k}}$ and $\overline{h_{t,dc}^{k}}$ satisfy

in which $h^{\text{cap}}$ is the maximum charge/discharge power of HES.

Based on the supply demand status in the building, each building can sell excess power or procure power from the grid. Therefore, the exchange power $g_{t}^{k}$ between the building and the grid can be depicted as follows,

4) Constraints: The action $A_{t}$ corresponding to the state $S_{t}$ should satisfy the following constraints,

. Constraint (9) denotes the total exchange power in a microgrid of buildings. Note that constraints (10) and (11) regulate the lower and upper bound of the total exchange power of the microgrid and the exchange power of the building to ensure transmission safety, respectively. Constraint (12) regulates that the required charging energy of each EV should not exceed the battery capacity. Constraint (13) constrains the remaining required charging energy should not exceed the maximum energy that can be supplied during the remaining parking time. This constraint is used to satisfy the driver’s charging demand. Constraint (14) denotes the load balance in each building.

5) Objective Function: As the responsibility of the microgrid operator controller is to optimize the operation cost during the entire periods, the following expected cumulative cost within a sliding window is chosen as the objective function considering the uncertain charging demand and output of DRE in the buildings, i.e.,

where $t_{0}$ denotes the current decision stage, $T_{w}$ denotes the sliding window, $\pi$ denotes the EV scheduling policy and $c_{t}(s_{t}^{k},a_{t}^{k})$ denotes the one-step cost incurred by taking action $a_{t}^{k}$ at state $s_{t}^{k}$ for the $k$th building which is defined as follow:

where $\omega_{t}$ denotes the electricity price.

Based on the proposed model above, the microgrid operator controller should minimize $J$ to find an optimal scheduling policy $\pi^{*}$ for each decision stage. However, due to the coupled constraint (10) and the large state space and action space of the problem, the exact optimal solution of the above model can rarely be derived[21]. In the next section, we will explore a event-based approach with gradient search to approximately solve the problem.

The proposed MDP model suffers large state space difficulty as it is a state-based model and its state space increases exponentially with the number of buildings and EVs. Therefore, we propose a event-based optimization (EBO) framework to solve the large state space difficulty. In the EBO framework, the model and the solution focus on the event which depicts the sets of state transition[22]. The decision is event-triggered which can save computation burden. When the event is defined approximately, EBO can be applied to solve MDPs with large state space[23]. Due to these advantages, EBO has been applied in various fields, such as HVAC control[23, 24], communication network[25], stock trading[26], etc.

In this paper, our idea to use EBO comes from the fact that it may not need to describe the detailed EV charging state and output of the DRE which incurs the large state space. Instead, we can use event to depict the elastic ratio of the EV charging, DRE and HES. The larger elastic ratio means the larger control margin during scheduling.

Based on this idea, we can firstly define the elastic ratio of EV charging in each building, i.e.,

where $\overline{n^{k}}$ denotes the number of charging piles in the $k$th building. Equation (17) describes the elastic degree of charging which can be delayed. Secondly, the elastic ratio of HES in each building can be defined as its SOC, i.e., $I_{t,HES}=b_{t}^{k}$. The larger $b_{t}^{k}$ means that the HES can provide a more flexible charge/discharge power for operation cost optimization. Thirdly, the elastic ratio of DRE can be defined as follows,

where $\overline{r^{k}}$ denotes the generation capacity of the DRE in the $k$th building. Equation (18) describes the generation level of the DRE and the higher level indicates the larger potential of operation cost saving.

Based on the elastic ratios introduced above, we can finally define the event as follows,

where

. In equation (19), $e_{t}^{k}$ denotes the triggered event in the $k$th building at stage $t$. The value of $e_{t}^{k}$ is within $[0,1]$. If we divide this interval equally with discrete unit set as 0.1, the event space for each building is limited as 10 which is largely reduced compared with the large state space in the proposed MDP model.

Another difficulty of the proposed MDP is the large action space. It is of great computation burden to compute the charge control variables $z_{t,i}^{k}$ for each EVs. Therefore, we propose a randomized parametric event-based policy to alleviate the large action space impact of this problem.

The charge control of each EV is implemented into two steps. Firstly, the microgrid operator controller decides a parametric charge ratio $\alpha_{t}^{k}$ for each building and is chosen as the event-based action, i.e., $a_{t}^{k}=\alpha_{t}^{k}\in[0,1]$. In this way, the total charge power for each building can be described as follows,

. As the charge ratio $\alpha_{t}^{k}\in[0,1]$, the action space of the proposed problem can be largely restricted. Based on [24], the performance of randomized policy may be better than deterministic policy and easier to obtain. Therefore, we will find a randomized parametric event-based policy $\sigma$ for the proposed problem, i.e.,

where $\mathcal{P}$ denotes a probability distribution over action space. Equation (22) means that the action $a_{t}^{k}$ will be chosen based on probability distribution $\mathcal{P}$ and observed event $e_{t}^{k}$. When an optimal randomized parametric event-based policy is obtained, the action with the highest probability can be selected for implementation in practice.

Secondly, after the microgrid operator controller allocates the charge ratio for each building, the charge controller in each building should decide which EV should be charged and keep the total number of charged EVs within $n_{t,m}^{k}+a_{t}^{k}(n_{t,c}^{k}-n_{t,m}^{k})$. Therefore, we use a modified least-laxity-longer-processing-time-first (mLLLP) principle to select EVs to charge, which is introduced in [27]. The mLLLP principle generates a complete order among EVs and selects EVs based on the remaining processing time $E_{t}^{i}/P$ and the EV laxity $T_{t}^{i}-E_{t}^{i}/P$.

Due to the existence of constraint (10), the proposed MDP is coupled among all the buildings in the microgrid. Therefore, in this paper we propose a constrained gradient-based policy optimization method to search the optimal randomized parametric event-based policy considering the coupled constraint.

First, we neglect constraint (10) and the proposed model can be decoupled into single building optimization problem. For each building, let $\sigma$ and $\nu$ denotes two event-based policies, the following state-based performance difference formula can be derived based on [28].

where $J_{t_{0}}^{k,\sigma}$ and $J_{t_{0}}^{k,\nu}$ denotes the value function from stage $t_{0}$ to $t_{0}+T_{w}-1$ corresponding to the event-based policy $\sigma$ and $\nu$ for the $k$th building, $\beta_{t}^{k,\sigma}$ is the state distribution at stage $t$ corresponding to $\sigma$, $P_{t}^{k,\sigma}$ and $P_{t}^{k,\nu}$ denotes the transition probability at stage $t$ corresponding to $\sigma$ and $\nu$, $c_{t}^{k,\sigma}$ and $c_{t}^{k,\nu}$ denotes the one-step cost at stage $t$ corresponding to $\sigma$ and $\nu$. Note that the initial distribution $\beta_{t_{0}}^{k,\sigma}$ is independent with policy ${\sigma}$ or ${\nu}$. By using recursion and $\beta_{t_{0}}^{k,\sigma}P_{t_{0}}^{\sigma}=\beta_{t_{0}+1}^{k,\sigma}$, the last equation can be obtained.

Based on (23), we can extend the performance difference for event-based optimization, i.e.,

where $\mathcal{S}$ denotes the state space. When policy $\nu$ is close to $\sigma$, the performance gradient at stage $t_{0}$ can be derived as follows by observing event $e_{t_{0}}^{k}$,

where $\sigma_{t_{0}}$ denotes the detailed charge control policy at stage $t_{0}$ for the $k$th building.

Note that policy $\sigma$ is a randomized policy which selects the action with specific probability. Suppose there are $M$ actions for the $k$th building at stage $t_{0}$ and each action is denoted as $\alpha_{t_{0}}^{k,m}$. Let $p_{t_{0}}^{k,m}$ denotes the probability to choose $\alpha_{t_{0}}^{k,m}$. Then there is the following relationship. The proof is given in the Appendix.

Substituting (26) and (27) into (25), the policy gradient can be finally obtained which is shown below,

where

denotes the incurred future total cost when taking action $\alpha_{t_{0}}^{k,m}$ for current state $s_{t_{0}}^{k}$ and $\frac{\partial J_{t_{0}}^{k,\sigma}(e_{t_{0}}^{k})}{\partial\sigma_{t_{0}}}=(\frac{\partial J_{t_{0}}^{k,\sigma}(e_{t_{0}}^{k})}{\partial p_{t_{0}}^{k,1}},\frac{\partial J_{t_{0}}^{k,\sigma}(e_{t_{0}}^{k})}{\partial p_{t_{0}}^{k,2}},...,\frac{\partial J_{t_{0}}^{k,\sigma}(e_{t_{0}}^{k})}{\partial p_{t_{0}}^{k,M}})$.

Based on the policy gradient (28), the randomized parametric event-based policy can be updated as follows during policy optimization,

where $\sigma_{t_{0},j}$ denotes the updated event-based policy at $j$th iteration, $\delta_{j}=1/(1+\xi j)$ denotes the update step at $j$th iteration and $\xi$ denotes the decay factor.

Due to the uncertainties in the DRE generation and EV charging demand, it is impractical to analytically compute equation (28) under expectation. Therefore, the Monte Carlo simulation method is adopted to estimate (28). The estimation algorithm is summarized in Algorithm 1.

As mentioned before, the derived policy gradient neglects the coupled constraint (10). In order to satisfy this transmission safety constraint, the following adjusting mechanism is proposed to ensure the feasibility of policy $\sigma$.

Adjusting Step I: If the total exchange power exceed the upper bound of (10) by $\Delta$, the total number of EVs to be charged should be reduced by $\Delta/P$. Each building should reduce the number of charged EVs by $(\partial J_{t_{0}}^{k,\sigma}\Delta/\partial\sigma_{t_{0}}^{k})/(P\sum_{k=1}^{K}\partial J_{t_{0}}^{k,\sigma}/\partial\sigma_{t_{0}}^{k})$. The policy $\sigma$ can be updated by solving the following Quadratic Programming problem,

where $p_{t_{0},j+1}^{k,i}$ denotes the selection probability in $\sigma_{t_{0},j+1}$.

Adjusting Step II: If the total exchange power is below the lower bound of (10) with $-\Delta$, the total number of EVs to be charged should be increased by $\Delta/P$. Each building should increase the number of charged EVs by $(1-\partial J_{t_{0}}^{k,\sigma}/\partial\sigma_{t_{0}}^{k}/\sum_{k=1}^{K}\partial J_{t_{0}}^{k,\sigma}/\partial\sigma_{t_{0}}^{k})\Delta/P$. Similarly, the policy $\sigma$ can be updated by solving the following problem,

Note that as $\partial J_{t_{0}}^{k,\sigma}/\partial\sigma_{t_{0}}^{k}$ can be considered as the marginal operation cost for the $k$th building, the proposed adjusting mechanism allocates the reduced or increased number of charged EVs for each building based on this marginal cost. When required to reduce the charge demand, the building with the large marginal cost should largely reduce its charged EVs. On the contrary, when required to increase the charge demand, the building with small marginal cost should largely increase its EVs to be charged. The motivation of solving (31) and (32) is to minimize the probability difference between the adjacent policy $\sigma_{t_{0},j+1}$ and $\sigma_{t_{0},j}$ while reduce or increase the expected charge ratio to satisfy the allocated reduced or increased charge demand for the $k$th building.

The idea of this adjust mechanism can be denoted as Fig. 2. As there exists large number of discrete variables and non-linear constraints, the feasible policy space can be considered as disconnected. When constraint (10) is not violated, the policy update happens within a feasible policy set. When violated, the policy update should transfer to another feasible policy set based on this adjust mechanism. In the end, the proposed constrained gradient-based policy optimization for the problem is summarized in Algorithm 2.

We take building load data from [29] as shown in Table I. In the experiment, we consider there are three buildings in the microgrid. The first two are residential buildings and the rest is office building. We take distributed wind generation data from [5] and set as the predicted value. The actual output of DRE in each building is assumed to follow normal distribution with predicted value as the mean value and $10\%$ of the predicted value as the standard deviation. Fig. 3 shows the realization of the actual output of DRE in three buildings. The time-of-use electricity price is shown in Table II.

We consider there are 200 EVs in the microgrid and their drivers live evenly in the two residential buildings and work in the office building. In the experiment, we assume that the departure time from residential and office buildings follow normal distribution $\mathcal{N}(7:00,60\text{min})$ and $\mathcal{N}(17:00,60\text{min})$, respectively. The trip time between building #1 (k=1) and building #3 (k=3) follows normal distribution $\mathcal{N}(60\text{min},30\text{min})$. The trip time between building #2 (k=2) and building #3 (k=3) follows normal distribution $\mathcal{N}(90\text{min},30\text{min})$.

The battery specification of the Nissan Leaf EV [30] is used in the experiments. The required charging energy $\eta_{t+1}^{i}$ of future parking event is sampled based on the probability distribution of the trip distance and the electric drive efficiency which is introduced in [5]. We take the parameters of HES from [31]. The detailed parameters are shown in Table III.

In the experiment, we consider this problem on a daily basis with $T=48$, $\Delta T=30$minutes and $T_{w}=12$. The event is evenly discretized by 0.1. The action selection probabilities of the initial policy $\sigma$ are all set as equal.

The scheduling results are shown in Fig. 4 which shows the optimized selection probability, total charging power, observed event and SOC of HES for three buildings at each stage. Note that Fig. 4(a), Fig. 4(b) and Fig. 4(c) shows the optimized selection probability of each charge ratio action in the randomized event-based policy after observing the event as shown in Fig. 4(d), Fig. 4(e) and Fig. 4(f). The darker color indicates higher selection probability. From Fig. 4(a), Fig. 4(b) and Fig. 4(c), it can be seen that the selection probabilities tend to achieve high value near the departure time, such as 7:00 in building #1 and building #2 and 17:00 in building #3. The reason why the charging probability is small at the arrival and becomes large at the departure lies in two aspects. The first is that parking deadline approaches and the charging demand should be satisfied. Another important reason is that the distributed wind power begins to increase during time interval (2:30-7:30) and (14:00-16:30). Note that as there are few EVs in building #1 and building #2 during (8:00-16:00), there is no action whose selection probability is significantly higher than others. The same reason holds for building #3 during (24:00-6:00). Furthermore, as there is no EV parked in building #3 after 20:00 considering time window $T_{w}$ set as 6 hours, the policy gradient keeps zero and the selection probability for each action remains unchanged and equal.

The total charging power and observed event is shown in Fig. 4(d), Fig. 4(e) and Fig. 4(f). Due to the stochastic charging demand of EVs and their distinct departure, the charging behavior occurs during (16:00-10:00) for building #1 and building #2 and during (10:00-18:00) for building #3. Furthermore, it can be found that the peak of the total charging power occurs later in building #2 than building #1 after 15:00. This is because that the trip time from building #3 to building #2 is longer than that from building #3 to building #1. Based on the delayed feature of the optimized selection probability in building #3, the peak occurrence of charging power are also delayed to 15:00 and gradually decrease due to the departure of EVs. It can also be found that the trend of the observed event is similar with the trend of the SOC of HES in each building as shown in Fig. 4(g), Fig. 4(h) and Fig. 4(i). This is because the SOC of HES is one of the main factor which influence the value of the observed event. The difference between the trend of observed event and SOC of HES is caused by the distributed wind power generation and EV charging elasticity. In Fig. 4(g), Fig. 4(h) and Fig. 4(i), the decreasing of SOC indicates that the HES provides power for balancing building load and EV charging load and the increasing of SOC indicates that the excess generation of distributed wind power. The small peak at 10:00 in building #2 is because the insufficient generation of distributed wind power and larger load demand comparing with building #1.

In order to analyze the performance of the optimized event-based EV charging policy, we compare the derived event-based policy with rule-based charging policy and ideal charging policy. The rule-based charging policy will satisfy the EV charging demand as soon as possible once connected to the charging pile in the building. The ideal charging policy is derived by implementing model predictive control (MPC) method with precise information of the EV charging demand and wind power generation and the same length of sliding window. Particularly, the optimal scheduling of HES are also considered as the control variables in MPC. In this way, it will introduce some non-linear constraints in the MPC model which should be linearized, such as the product between integer variables and continuous variables. The performance of the above three policies are shown in Table IV. It can be seen that the rule-based policy achieves highest operation cost as the EV charging control has no relationship with the building load and supply. On the contrary, the ideal charging achieves lowest operation cost. However, this policy can not be implemented in practice due to its requirement of seeing the future. Compared the event-based policy with ideal charging, it can be found that the performance of our policy is close to the idea policy and better than the rule-based charging policy. This demonstrates the effectiveness of the proposed EV charging control method.

The total exchange power of microgrid under the above three policies are also shown in Fig. 5. The peak of the total exchange power at 19:00 is caused by the high building load in building #1 and building #2 at night. It can be seen that the rule-based charging policy exceed the maximum transmission power $\overline{G}=5600\text{kW}$ while the peak of the latter two policies are below this upper bound. This indicates that the proposed constrained gradient-based policy optimization method can ensure the transmission safety. It can be seen from the figure that the main reason why the event-based charging and ideal charging outperform the rule-based charging lies in two aspects. The first is the total exchange power of the event-based policy and ideal policy postpones during (9:00-12:00) to enjoy low electricity price after 11:00. The second is the ahead of schedule of the event-based policy and ideal policy during (13:00-22:00) to avoid high electricity price during (19:00-22:00) and enjoy free wind power generation around 15:00.

Lastly, it is important to investigate the convergence rate of the proposed constrained gradient-based policy optimization. Therefore, Fig. 6 shows the total iteration number at each decision stage. For this experiment, the minimum iteration number is 5 which happens at 7:30 and the maximum iteration number is 50 which happens at 16:30 and 19:30. It takes about 25.3 iterations in average to find the optimal event-based charging policy at each decision stage. The average computation time at each decision stage is 5.2 minutes under the simulation environment i7-11700K@3.60GHz. This indicates the proposed algorithm requires a handful of iterations and its running time can be acceptable in practice.