Personalized Dynamic Pricing Policy for Electric Vehicles: Reinforcement learning approach

There has been a considerable amount of research recently to shed some insight into the decision-making problem. The following studies have been focusing on power grid stability and incorporating pricing from a network balancing perspective. In order to achieve this, we modeled the EV user’s behavior with different levels of accuracy. In the following, we hence outline them as price-responsive algorithms with the following attributes: i) optimization/solution method; ii) detailedness of EV models. Most of the methods simplify the EV user’s decision and propose multilayer, centralized, or distributed (and often game-theoretic) solutions.

In [14], the authors considered both the charging station pricing problem and the charging station selection problem using a two-stage Stackelberg model. The authors assumed the same state of charge for the EVs (i.e., same charging cost, same driving range). In [15], the authors proposed a fast-EVCS pricing strategy for the voltage control of electricity distribution networks using a double-layer optimization model to minimize the total voltage magnitude deviation of distribution networks without loss of income. The authors could not consider the user’s willingness, which determines the demand for the electricity distribution network. In [16], the authors proposed the joint admission and pricing approach to maximize a charging station’s profit, which also allows them to find a trade-off between the revenue of the charging station and the EV users’ satisfaction (waiting time). The authors assumed that a charging station makes admission decisions for homogeneous EV users’ charging (i.e., the charging station can refuse some EV users to charge to maximize their profit). In [17], pricing mechanisms were proposed under non-cooperative and cooperative scenarios, respectively, for the evenly distributed charging load with the assumption that the charging decisions were made by an aggregator. In [18], a single-leader-multi-follower Stackelberg game is proposed, in which the charging network operator acts as the leader and the EV customers are the followers. The leader optimizes the prices of charging stations, and each follower makes a selection between the nearest charging station and a less busy one. For maintaining power grid stability, in [19], the authors proposed the customer allocation algorithm for maximizing their utility based on the waiting time. In [20], the authors studied the pricing of charging stations using a non-cooperative oligopoly game with assumptions such as homogeneous charging stations, homogeneous electric vehicles, and no road system constraints. In [21], the authors resolved the charging station selection problem based on the electricity price offered by the charging station, its distance from the charging station, and EV users’ preferences for a certain charging station. They assumed that the demand for each charging station increases as its charging price decreases. In [22], the authors resolved the overlapping issues that occur at peak time by dynamically adjusting the prices of the charging stations. Here, it is assumed that the charging stations cooperatively determine the prices. In [23], the authors conducted a survey to understand how electric vehicle drivers respond to uncertain future prices when they charge their vehicles away from home. The motivation of the authors is that dynamic pricing of electricity can help operators spread the demand for electricity to avoid costly infrastructure investments. The results suggest that electric vehicle drivers are generally risk-averse when it comes to uncertain pricing, and there is a non-linearity in their choices. In [24], the authors proposed a centralized EV charging-control model that schedules the charging of EVs. Through the case study, they showed the benefits of the model in reducing charging costs, negative impacts on the distribution system, and unserved EV charging demand compared to simpler heuristics.

Finally, in [25], the authors proposed dynamic demand-responsive price adjustment schemes based on charging station queue lengths. The results showed that these schemes were effective in balancing charging demand across fast-charging stations, reducing average waiting time, increasing charging station revenue, and improving social welfare. As indicated in [25], using more sophisticated charging and EV user models in combination with scalable price shaping algorithms is of capital importance. In this line of thought, data-dependable methods may bring new aspects to naturally integrating model parameter changes into price decision algorithms.

Electric vehicle charging stations are categorized as follows: 1) Level-1 chargers, which are often used at home, use 120 V alternating current (AC). Unlike other chargers, level-1 chargers do not require the installation of any additional equipment; 2) Level-2 chargers, which are often used for both residential and commercial EVCSs, use a 240 V (for residential) or 208 V (for commercial) plug; and 3) Level-3 chargers, also known as direct current (DC) fast chargers, are only used in commercial and industrial applications.

The main aim of this study is to investigate this highly interdependent and complex decision-making problem, in which all stakeholders make self-interested decisions. In this way, we will focus on demand reponsive pricing algorithms (in contrast to the price-responsible approaches presented in Section 1.1.) with detailed EV models. We propose a 1-dimensional highway simulation environment that is dynamically interacting and includes more practical factors, such as estimated state of charge (driving distance to empty), the number of available charging slots, the self-interested decision-making of all stakeholders, non-cooperative decision-making, and a monetary utility function, than the existing studies [14, 16, 18, 22]. Then, we propose a personalized dynamic pricing policy (PeDP) for fast-EVCSs using a reinforcement learning (RL) approach [36], since it is highly expected that fast-EVCSs might apply pricing policies using artificial intelligence (AI) to maximize their revenue because the charging environment is highly coupled with many agents’ decisions, competitors pricing policies, government policies, etc., which are hard to expect/predict. In such an environment, it is hard to utilize rule-based approaches or optimization-based approaches. It is assumed that the EV user’s privacy is already protected and available based on a technique discussed in [32, 33, 34]. As shown in Figure 1, EV users transfer information on their estimated state of charge (eSOC, privacy-preserved information, i.e., the information is only shared with the fast-EVCS) to the fast-EVCSs while they are on the highway. An AI fast-EVCS dynamically adjusts the personalized electricity price (action) that is offered to the corresponding EV user, in which the price is affected by public and/or privacy-preserved information from the competing fast-EVCSs and the EV users, respectively (environment). Each self-interested EV user updates its charging station selections based on the offered personalized electricity price and other given information (eSOC, available charging slots), in which a game-theoretical approach is applied to model the selfish behavior of the EV users. When the EV users arrive at the AI fast-EVCS for charging, the AI fast-EVCS charges an electricity bill to the EV users (reward). We conduct numerical simulations in the proposed environment with fast-EVCSs and dynamically arriving EV users in order to 1) figure out the systemic differences regarding the waiting time between the petrol stations and fast-EVCSs, the proposed PeDP is compared with the classic Bertrand competition model (from the system perspective); 2) evaluate the performance of the proposed PeDP and analyze the importance of the information on the policy (from the fast-EVCS perspective); and 3) show that sharing the privacy-preserved information can be misused by AI-based PeDP in a certain situation in the EV charging market (from the EV user perspective).

Our main contributions are summarized as follows.

• •

  A system model: The proposed environment considers practical factors such as heterogeneous EVs (e.g., EV-dependent charging time, SOC (travel range)) and dynamically changing capability of the fast-EVCSs, where the parameters for the EVs and fast-EVCSs are based on the public performance data [37] (Q3-1). We model the charging station selection problem as an anonymous hedonic game [38, 39], which is suitable for modeling the situation where the EV users greedily select fast-EVCSs based on their utility function. Here, we propose a monetary utility function for EV users by transforming the expected waiting time into money using the value of travel time (VTT) [40]. To the best of our knowledge, this is the first work that proposes a flexible and extensible EV charging market simulation environment for the personalized dynamic pricing problem.

• •

  Personalized dynamic pricing policy: Due to the dynamical and highly interdependent interactions between EV users and fast-EVCSs in the proposed environment, it might not be easy to apply the existing approaches, such as the Stackelberg game model [41]. We propose the PeDP using the Q-learning algorithm that interacts with the proposed environment. Different from existing pricing literature, the proposed approach finds the PeDP of the fast-EVCS according to the dynamic environment instead of finding the price equilibrium in the static environment.

Finally, the above-mentioned methodological solutions have been tested in a numerical simulation context, demonstrating the benefits of dynamic price shaping. As such, we evaluate the PeDP and investigate the importance of information usage in the EV charging market. Firstly, we show the importance of the waiting time in the EV charging market by comparing the PeDP and the classic Bertrand model. Secondly, we evaluate the performance of the PeDP and investigate the effect of the public and/or privacy-preserved information by varying the observable states for the AI fast-EVCS from the service provider perspective. Finally, we present a privacy-preserved information misuse case by the AI-based PeDP in a certain geographical condition.

The remainder of this paper is organized as follows: Section 2 presents the EV charging market simulation environment, which is used as the environment for the model-free reinforcement learning approach. In Section 3, we propose a charging station selection game (CSSG) model that is a key part of the environment. In Section 4, we propose a PeDP algorithm using a model-free RL approach. Numerical simulations are provided in Section 5. Finally, conclusions and future work are presented in Section 6.

The time taken to complete a full charge is longer compared to the initial phase of charging, which is between 0% and 80%, approximately as shown in Figure 2 [46]. For considering a rational charging strategy due to these charging characteristics, it is assumed that EV users charge up to 80% of their full SOC. Also, it is assumed that there are no uncertainties in the travel range of the EVs. Therefore, we define the estimated state of charge $eSOC_{(i,j)}$ of EV user $b_{i}$ arrival at fast-EVCS $e_{j}$ as follows:

where $dist(i,j)$ and $k_{\text{eff}}$ are the euclidean distance between EV user $b_{i}$ and fast-EVCS $e_{j}$, and energy efficiency that is assumed as a constant ($k_{\text{eff}}=800$[km/SOC]) in this study, respectively. Note that it is assumed that the speed of each EV is constant ($100$[km/h]), and factors that impact the energy efficiency, such as temperatures, aging effects, etc., are constant.

The CSSG is solved using a decentralized algorithm [47]. Each time step, which is the same update frequency as the fast-EVCS status update in this study, each EV user makes a decision to maximize their monetary utility. EV users’ competitive and strategic behaviors can be modeled from game-theoretic perspectives. Here, it is assumed that each EV user is able to access information from the fast-EVCSs, such as locations, electricity prices, and the number of available charging slots. If every EV user is satisfied with their charging station selection, it can be considered that they are mutual-exclusively grouped³³3For the sake of convenience, we use the term “group” that only indicates the EV users who select the same fast-EVCS. for each of the fast-EVCSs. This status can be described as the concept called Nash stable partition, which is denoted by $\mathbf{P}$. A Nash stable partition always exists and is computed by Algorithm 2. More details of the algorithm are given in A.

The fast-EVCSs compete with each other to maximize their revenue by adjusting their personalized electricity prices dynamically. The revenue of each fast-EVCSs is determined by the number of visited EVs and the amount of electricity they charge. Here, we assume that other operating costs of charging stations are fixed and not included in mitigating other effects. Therefore, the goal of the fast-EVCS is to perform personalized dynamic pricing that maximizes its revenue as follows:

Charging station operators will be able to access a wide variety of information. Also, the charging environment includes many uncertainties, especially due to the drivers’ strategic charging decisions. The Reinforcement Learning (RL) approach is known as one of the suitable approaches in an unknown environment with uncertainties where ordinary dynamic programming is not scalable. Reinforcement Learning (RL) is suitable for addressing complex decision-making problems [48], [49]. We consider the fast-EVCS revenue maximization problem in which an agent (AI fast-EVCS, $e_{\bar{j}}$) interacts with the proposed environment in a sequence of actions, observations, and rewards. As shown in Figure 3, the AI fast-EVCS serves as the agent; competing fast-EVCSs and strategically behaving EV users are the environment; adjusting personalized electricity prices of the AI fast-EVCS denotes the action that the AI fast-EVCS offers to each corresponding EV user; the public electricity price and the number of available charging slots of the competing fast-EVCSs, and the privacy-preserved information of the EV users represent the state; and the AI fast-EVCS revenue is a reward.

In this paper, the personalized dynamic pricing problem is modeled as a Markov decision process (MDP) because it is a decision-making problem in an unknown stochastic environment. In this MDP model, an agent selects an action to execute with respect to the current state based on its policy. The agent receives an immediate reward and transitions to the next state, in which the reward depends only on EV users’ charging station decisions and their SOC in this study. The objective of reinforcement learning is to find the policy that maximizes the expected discounted return

The learning process is presented in Algorithm 1, First, we initialize the parameters of the Q-network with $\theta$. As the charging station operates 24/7, the algorithm runs until the convergence criterion is met. Note that the convergence time might be influenced by the parameters of the simulation environments (e.g., EVUs rate, the number of fast-EVCS on the highway, etc.). In most of the cases in this study, it takes 15-30 simulation days to converge. In practical situations, fast-EVCS owners who want to apply the proposed approach might need either a simulation environment or some reference data that applies personalized pricing for an initial PeDP. At each time step, the AI fast-EVCS adjusts the personalized electricity price based on the $\epsilon$-greedy strategy and offers the price to the corresponding EVU. Accordingly, the immediate reward and the next observation are obtained from the environment. The transition is stored sequentially. To update the Q-network, we randomly sample a sequence of transitions $\big{\langle}S_{t},A_{t},R_{t},S_{t+1}\big{\rangle}$.

The entire process of the proposed approach is presented in Figure 4. First, we initialize the environment using Algorithm 2, replay memory, and Q-network weights $\theta$. We adopt the replay memory (experience replay) to have the advantages of improving learning speed and mitigating the undesirable temporal correlations [51]. Once the AI fast-EVCS starts to apply the proposed approach, it receives public and privacy-preserved information from competing fast-EVCSs and EV users, respectively. Then, the AI fast-EVCS $e_{\hat{j}}$ adjusts personalized electricity price $p^{t}_{(i,\bar{j})}$ to corresponding EV user $b_{i}$ using the $\epsilon-$greedy policy, which randomly chooses an action with probability $\epsilon$ and most of behaves in a greedy manner. We set $\epsilon$ as 0.2, and its discounting factor is 0.99 for all numerical experiments in this study. It offers the price $p^{t}_{(i,\bar{j})}$ to each corresponding EV user. The environment, which includes EV users’ charging station selections and competing fast-EVCSs public electricity prices and the number of available charging slots, is updated using Algorithm 2. It observes individual revenue from each EV user and updates pricing policies: 1) Yes (arrival and charging) - update pricing policy; 2) Not arrived - no update; and 3) No (pass by the charging station) - update pricing policy. During the training stage, the pricing policy of the AI fast-EVCS is trained only when EV user $b_{i}$ is charging at the AI fast-EVCS or when it has passed the AI fast-EVCS. To avoid excessive training time, this process runs until the criterion is met, $\Delta L=|L^{t}(\theta)-L^{t-1}(\theta)|\leq\lambda$ where $\lambda$ (=0.005) is a stopping criterion, in which the criterion is the loss difference between the last two iterations. Finally, the AI fast-EVCS will obtain the PeDP.

In the EV charging market, not only the electricity price but also the expected waiting time play a very important role in the EV user’s charging station decision. In this numerical simulation, we show the importance of the expected waiting time by comparing the following three cases. First, we show the conventional fast-EVCSs pricing competition using the classic Bertrand competition model in Appendix B: Algorithm for the classic Bertrand competition model. Second, we show the pricing competition in the duopoly market using the proposed PuDP, where a utility function of the EV users is Eq.(8) (i.e., the EV users only consider the electricity price).

Due to the long charging time of EVs, the EV charging market has been expected to be different from the petrol station market. From the three comparison simulation results, it is found that the expected waiting time plays a very important role in the pricing policy of the fast-EVCSs. Additionally, we find that the PuDP without expected waiting time converges to the same Nash equilibrium price as the classic Bertrand competition model’s Nash equilibrium price in the duopoly EV charging market.

The main outputs of the simulation are the revenue and the number of charged EV users for each fast-EVCS. Figure 10(a) and (b) show the accumulative revenue and the number of charged EVs at each fast-EVCS. Figure 10(a) and (b) clearly show that the AI-fast EVCS $e_{2}$ with the PeDP makes the most revenue among them by attracting most of the EV users. Figure 10(c) and (d) show that the EV users who select $e_{2}$ pay less electricity cost but cost more on waiting time than the EV users who select either $e_{1}$ or $e_{3}$. From Figure 11, we can see the more detailed PeDP. Figure 11(a) shows the trajectories of the number of available charging slots and the personalized electricity price over time for each fast-EVCS based on their PeDP. Based on the public/privacy-preserved information, such as the number of available charging slots, the public electricity prices of competitors, etc., the AI fast-EVCS adjusts the personalized electricity price and offers it to the corresponding EV user. Each EV user selects one fast-EVCS among them to charge based on the public or personalized electricity prices and the number of available charging slots of the fast-EVCSs, as shown in Figure 11(b). In other words, the EV users choose fast-EVCSs based on the proposed monetary utility function, Eq.(1). Figure 11(c) shows the monetary utility of the EV users for the fast-EVCS $e_{2}$. In Figure 11(b), for example, let us see the decision-making process of the EV user, $b_{i}$, who enters the highway at 197 minutes with SOC level 0.67 and visits the AI fast-EVCS, $e_{2}$, at 319 minutes with SOC level 0.43. As shown in Figure 11(a) the lower personalized electricity price of $e_{2}$ attracts the EV user despite the lack of available charging slots at the fast-EVCS $e_{2}$. Exceptionally, even with the lower personalized electricity price and no waiting cost between 244 minutes and 258 minutes, the EV user selects $e_{3}$, which is 200km further away from $e_{2}$. This is due to the lack of available charging slots at $e_{2}$. We can see a small decrease in the total monetary utility value in Figure 11(c). In other words, EV users always try to make economical choices, and the AI-fast EVCS offers marginal prices to attract EV users.

Case 2.1 shows the lowest revenue while the number of charged EVs is high, as shown in Table 4. Because of the limited information, the AI fast-EVCS $e_{1}$ conservatively adjusts the public electricity price. For example, the AI fast-EVCS $e_{1}$ somehow assumes that the number of available charging slots $\Delta c_{2}^{t}$ is always the maximum value because the AI fast-EVCS $e_{1}$ does not observe $\Delta c_{2}^{t}$ for the pricing policy.

In Case 2.2, the AI fast-EVCS $e_{1}$ receives the privacy-preserved information $v^{t}_{\mathbf{P}}$, which is $eSOC_{(i,1)}$, $\forall b_{i}\in\mathcal{B}$. Based on the information, the AI fast-EVCS adjusts the personalized electricity price for each corresponding EV user. This case shows the advantage of observing the privacy-preserved information. The AI fast-EVCS can implicitly consider the fixed cost $c_{\text{f}}$ in Eq.(1) because the cost depends on the $\Delta SOC_{(i,j)}$. For example, the monetary utility of EV user $b_{i}$ is higher than that of EV user $b_{i^{\prime}}$ when $eSOC_{(i,1)}>eSOC_{({i^{\prime}},1)}$. Therefore, based on the information, the AI fast-EVCS determines personalized electricity prices to attract customers with 12.9% more revenue than Case 2.1.

Case 2.3 shows the simulation results when the AI fast-EVCS $e_{1}$ considers the number of available charging slots $\Delta c_{2}^{t}$ and the public electricity price $p^{t}_{(\cdot,2)}$ as the state. $\Delta c_{2}^{t}$ is directly related to the expected waiting time at $e_{2}$ which affects the EV user’s decision-making. In other words, if the public electricity price gap between two fast-EVCSs, $p^{t}_{(\cdot,2)}-p^{t}_{(\cdot,1)}$, is bigger than the waiting cost at $e_{1}$, which is the latter term of Eq.(1), the EV user selects the AI fast-EVCS $e_{1}$ for charging. Table 4 shows that Case 2.3 yields 53.12% and 35.63% higher returns than Case 2.1 and Case 2.2, respectively. In other words, to increase revenue, it is shown that it is more important to utilize the number of charging slots $\Delta c_{2}^{t}$ rather than utilizing other information.

In Case 2.4, finally, the AI fast-EVCS $e_{1}$ considers the public electricity price $p^{t}_{(\cdot,2)}$ and the number of available charging slots $\Delta c_{2}^{t}$ and estimated SOC $eSOC_{(i,1)},\forall b_{i}\in\mathcal{B}$ as the state. The AI fast-EVCS has the highest revenue among the other cases, even if the AI fast-EVCS loses some EV users. Nevertheless, the reason for the high revenue can be found in the electricity utility in Table 4. In Case 2.4, the AI fast-EVCS benefits from EV users by offering higher personalized electricity prices than in any other case. Note that the purpose of the simulation results is not to show a quantitative comparison but to analyze the importance of the information as the states of the AI fast-EVCS.

In the previous two cases, we showed the importance of the information and the performance of the proposed approach. In Case 3, we present a case where even the privacy-preserved information could be abused by the AI fast-EVCS. In the duopoly EV charging market, when two fast-EVCSs are in different locations, the AI fast-EVCS close to the entrance of the highway is likely to misuse the privacy-preserved information for its revenue improvement. We show a misuse case in the duopoly EV charging market using the PeDP in the environment, as shown in Figure 12. Except for the fast-EVCS locations ($e_{1}=$200km and $e_{2}=$400km), the environment is the same as the environment for Case 2.2.

As similar to Case 2.2, the AI fast-EVCS attracts most of the EV users with marginal peronalized electricity prices. However, due to the low SOC, some EV users cannot reach $e_{2}$. In other words, these EV users are forced to charge at $e_{1}$ regardless of the electricity price. In such a situation where two fast-EVCSs are separated from each other, simply sharing eSOC, which is privacy-preserved information, can lead to an unfair pricing policy (i.e., causing a transformation from the duopoly market to the monopoly market). The misused PeDP obtained using the proposed approach for the environment is shown in Figure 13(a). As we discussed, the reason for the dramatically increasing personalized electricity price is obvious. Since the EV users do not have enough SOC to reach $e_{2}$, they only have the option to charge at $e_{1}$. The AI fast-EVCS empirically knows the facts and utilizes them. On the other hand, Figure 13(b) shows the PeDP of Case 2.2, which has no abuse of the privacy-preserved information. Therefore, EV users are always in the duopoly market when two fast-EVCSs are located in the same place. From the simulation results, we conclude that sharing the privacy-preserved information, which is the eSOCs of anonymous EVs, could implicitly change the duopoly market to the monopoly market and could be misused by the AI fast-EVCS. Therefore, in order to avoid misuse cases, policymakers should be more careful in permitting the utilization of privacy-preserved information or in the placement of fast-EVCSs such as Figure 9 (i.e., the oligopoly market).