Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges

$\mathcal{A}$, $a$

Action space, action.

$\mathcal{E}$

$\subseteq\mathcal{N}\times\mathcal{N}$, the set of lines connecting buses.

$J$

Expected total discounted reward.

$\mathcal{N}$

$:=\{1,\cdots,N\}$, the set of buses in a power network or the set of agents.

$\text{NN}(x;w)$

Neural network with input $x$ and parameter $w$.

$o$

Observation.

$\mathbb{P}$

Transition probability.

$Q_{\pi}$

$Q$-function (or $Q$-value) under policy $\pi$.

$r$

Reward.

$\mathcal{S}$, $s$

State space, state.

$\mathcal{T}$

$:=\{0,1,\cdots,T\}$, the discrete time horizon.

$\gamma$

Discounting factor.

$\Delta(\mathcal{A})$

The set of probability distributions over set $\mathcal{A}$.

$\Delta t$

The time interval in $\mathcal{T}$.

$\pi$, $\pi^{*}$

Policy, optimal policy.

A3C

Asynchronous Advantaged Actor Critic.

ACE

Area Control Error.

AMI

Advanced Metering Infrastructure.

ANN

Artificial Neural Network.

DDPG

Deep Deterministic Policy Gradient.

DER

Distributed Energy Resource.

DP

Dynamic Programming.

(D)RL

(Deep) Reinforcement Learning.

DQN

Deep $Q$ Network.

EMS

Energy Management System.

EV

Electric Vehicle.

FR

Frequency Regulation.

HVAC

Heating, Ventilation, and Air Conditioning.

IES

Integrated Energy System.

LSPI

Least-Squares Policy Iteration.

LSTM

Long-Short Term Memory.

MDP

Markov Decision Process.

OLTC

On-Load Tap Changing Transformer.

OPF

Optimal Power Flow.

PMU

Phasor Measurement Unit.

PV

Photovoltaic.

SAC

Soft Actor Critic.

SCADA

Supervisory Control and Data Acquisition.

SVC

Static Var (Reactive Power) Compensator.

TD

Temporal Difference.

UCRL

Upper Confidence Reinforcement Learning.

RL is a branch of machine learning concerned with how an agent makes sequential decisions in an uncertain environment to maximize the cumulative reward. Mathematically, the decision-making problem is modeled as a Markov Decision Process (MDP), which is defined by state space $\mathcal{S}$, action space $\mathcal{A}$, the transition probability function $\mathbb{P}(\cdot|s,a):\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ that maps a state-action pair $(s,a)\in\mathcal{S}\times\mathcal{A}$ to a distribution on the state space, and lastly the reward function $r(s,a):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$¹¹1 A generic reward function is given by $r(s,a,s^{\prime})$ where the next state $s^{\prime}$ is also included as an argument, but there is no essential difference between the case with $r(s,a)$ and the case with $r(s,a,s^{\prime})$ in algorithms and results. By marginalizing over next states $s^{\prime}$ according to the transition function $\mathbb{P}(s^{\prime}|s,a)$, one can simply convert $r(s,a,s^{\prime})$ to $r(s,a)$ [2].. The state space $\mathcal{S}$ and action space $\mathcal{A}$ can be either discrete or continuous. To simplify discussion, we focus on the discrete case below.

As illustrated in Fig. 2, in an MDP setting, the environment starts with an initial state $s_{0}\in\mathcal{S}$. At each time $t\!=\!\{0,1,\cdots\}$, given current state $s_{t}\in\mathcal{S}$, the agent chooses action $a_{t}\in\mathcal{A}$ and receives reward $r(s_{t},a_{t})$ that depends on the current state-action pair $(s_{t},a_{t})$, after which the next state $s_{t+1}$ is randomly generated from the transition probability $\mathbb{P}(s_{t+1}|s_{t},a_{t})$. A policy $\pi(a|s)\in\Delta(\mathcal{A})$ for the agent is a map from the state $s$ to a distribution on the action space $\mathcal{A}$, which rules what action to take given a certain state $s$.²²2$a\sim\pi(\cdot|s)$ is a stochastic policy, and it becomes a deterministic policy $a=\pi(s)$ when the probability distribution $\pi(\cdot|s)$ is a singleton for all $s$. The agent aims to find an optimal policy $\pi^{*}$ (may not be unique) that maximizes the expected infinite horizon discounted reward $J(\pi)$:

where the first expectation means that $s_{0}$ is drawn from an initial state distribution $\mu_{0}$, and the second expectation means that the action $a_{t}$ is taken according to the policy $\pi(\cdot|s_{t})$. Parameter $\gamma\in(0,1)$ is the discounting factor that penalizes the rewards in the future.

In the MDP framework, the so-called “model” specifically refers to the reward function $r$ and the transition probability $\mathbb{P}$. Accordingly, it leads to two different problem settings:

• •

  When the model is known, one can directly solve for an optimal policy $\pi^{*}$ by Dynamic Programming (DP) [12].

• •

  When the model is unknown, the agent learns an optimal policy $\pi^{*}$ based on the past observations from interacting with the environment, which is the problem of RL.

Since DP lays the foundation for RL algorithms, we first consider the case with a known model and introduce the basic ideas of finding an optimal policy $\pi^{*}$ with respect to (1). The crux is the concept of $Q$-function together with the Bellman Equation. The $Q$-function $Q_{\pi}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ for a given policy $\pi$ is defined as

which is the expected cumulative reward when the initial state is $s$, the initial action is $a$, and all the subsequent actions are chosen according to policy $\pi$. The $Q$-function $Q_{\pi}$ satisfies the following Bellman Equation: $\forall(s,a)\in\mathcal{S}\times\mathcal{A}$,

where the expectation denotes that the next state $s^{\prime}$ is drawn from $\mathbb{P}(\cdot|s,a)$, and the next action $a^{\prime}$ is drawn from $\pi(\cdot|s^{\prime})$. Here, it is helpful to think of the $Q$-function as a large table or vector filled with $Q$-values $Q_{\pi}(s,a)$. The Bellman Equation (3) indicates a recursive relation that each $Q$-value equals the immediate reward plus the discounted future value. Computing the $Q$-function for a given policy $\pi$ is called policy evaluation, which can be done by simply solving a set of linear equations when the model, i.e., $\mathbb{P}$ and $r$, is known.

The $Q$-function associated with an optimal policy $\pi^{*}$ for (1) is called an optimal $Q$-function and denoted as $Q^{*}$. The key to find $\pi^{*}$ is that the optimal $Q$-function must be the unique solution to the Bellman Optimality Equation (4): $\forall(s,a)\in\mathcal{S}\times\mathcal{A}$,

Interested readers are referred to textbook [12, Sec. 1.2] on why this is true. Based on the Bellman Optimality Equation (4), the optimal $Q$-function $Q^{*}$ and an optimal policy $\pi^{*}$ can be solved using DP or linear programming [12, Sec. 2.4]. In particular, policy iteration and value iteration are two classic DP algorithms. See [2, Chapter 4] for details.

This subsection considers the RL setting when the environment model is unknown, and presents classical RL algorithms for finding the optimal policy $\pi^{*}$. As shown in Fig. 1, RL algorithms can be divided into two categories, i.e., model-based and model-free. “Model-based” refers to the RL algorithms that explicitly estimate and online update an environment model from past observations and make decisions based on this model [16], such as upper confidence RL (UCRL) [17] and Thompson sampling [18]. In contrast, “model-free” means that the RL algorithms directly search for optimal policies without estimating the environment model. Model-free RL algorithms are mainly categorized into two types, i.e., value-based and policy-based. Generally, value-based methods are preferred for modest-scale RL problems with finite state/action space as they do not assume a policy class and have a strong convergence guarantee. The convergence of value-based methods to an optimal $Q$-function in the tabular setting (without function approximation) was proven back in the 1990s [19]. In contrast, policy-based methods are more efficient for problems with high dimensions or continuous action/state space. But they are known to suffer from various convergence issues, e.g., local optimum, high variance, etc. The convergence of policy-based methods with restricted policy classes to the global optimum has been shown in a recent work [20] under the tabular policy parameterization.

Before presenting classical RL methods, we introduce a key algorithm, Temporal-Difference (TD) learning [21], for policy evaluation when the model is unknown. TD learning is central to both value-based and policy-based RL algorithms. It learns the $Q$-function $Q_{\pi}$ for a given policy $\pi$ from episodes of experience. Here, an “episode” refers to a state-action-reward trajectory over time $(s_{0},a_{0},r_{0},s_{1},a_{1},r_{1},\cdots)$ until terminated. Specifically, TD learning maintains a $Q$-function $Q(s,a)$ for all state-action pairs $(s,a)\in\mathcal{S}\times\mathcal{A}$ and updates it upon a new observation $(r_{t},s_{t+1},a_{t+1})$ by

where $\alpha$ is the step size. Readers might observe that the second term in (5) is very similar to the Bellman Equation (3), which is exactly the rationale behind TD learning. Essentially, TD learning (5) is a stochastic approximation scheme for solving the Bellman Equation (3) [22], and can be shown to converge to the true $Q_{\pi}$ under mild assumptions [21, 23].

1) Value-based RL algorithms directly learn the optimal $Q$-function $Q^{*}$, and the optimal (deterministic) policy $\pi^{*}$ is a byproduct that can be retrieved by acting greedily, i.e., $\pi^{*}(s)=\arg\max_{a\in\mathcal{A}}Q^{*}(s,a)$. Among many, $\bm{Q}$-learning [24, 19] is perhaps the most popular value-based RL algorithm. Similar to TD-learning, $Q$-learning maintains a $Q$-function and updates it towards the optimal $Q$-function based on episodes of experience. Specifically, at each time $t$, given current state $s_{t}$, the agent chooses action $a_{t}$ according to a certain behavior policy.³³3Such a behavior policy, also called exploratory policy, can be arbitrary as long as it visits all the states and actions sufficiently often. Upon observing the outcome $(r_{t},s_{t+1})$, $Q$-learning updates the $Q$-function by

The rationale behind (6) is that the $Q$-learning algorithm (6) is essentially a stochastic approximation scheme for solving the Bellman Optimality Equation (4), and one can show the convergence to $Q^{*}$ under mild assumptions [19, 25].

SARSA⁴⁴4In some literature, SARSA is also called $Q$-learning. [26] is another classical value-based RL algorithm, whose name comes from the experience sequence $(s,a,r,s^{\prime},a^{\prime})$. SARSA is actually an on-policy variant of $Q$-learning. The major difference is that SARSA takes actions according to the target policy (typically $\epsilon$-greedy based on the current $Q$-function) rather than any arbitrary behavior policy in $Q$-learning. The following remark distinguishes and compares “on-policy” and “off-policy” RL algorithms.

2) Policy-based RL algorithms restrict the optimal policy search to a policy class that is parameterized as $\pi_{\theta}$ with the parameter $\theta\in\Theta\subseteq\mathbb{R}^{K}$. With this parameterization, the objective (1) can be rewritten as a function of the policy parameter, i.e., $J(\theta)$, and the RL problem is reformulated as an optimization problem (7) that aims to find the optimal $\theta^{*}$:

To solve (7), a straightforward idea is to employ the gradient ascent method, i.e., $\theta\leftarrow\theta+\eta\nabla J(\theta)$, where $\eta$ is the step size. However, computing the gradient $\nabla J(\theta)$ was supposed to be intrinsically hard as the environment model is unknown. Policy Gradient Theorem [28] is a big breakthrough in addressing the gradient computation issue. This theorem shows that the policy gradient $\nabla J(\theta)$ can be simply expressed as

Here, $\mu_{\theta}(s)\in\Delta(\mathcal{S})$ is the on-policy state distribution [2, Chapter 9.2], which denotes the fraction of time steps spent in each state $s\in\mathcal{S}$. Equation (8) is for a stochastic policy $a\sim\pi_{\theta}(\cdot|s)$, while the version of policy gradient theorem for a deterministic policy $a=\pi_{\theta}(s)$ [29] will be discussed later.

The policy gradient theorem provides a highway to estimate the gradient $\nabla J(\theta)$, which lays the foundation for policy-based RL algorithms. In particular, the actor-critic algorithm is a prominent and widely used architecture based on policy gradient. It comprises two eponymous components: 1) the “critic” is to estimate the $Q$-function $Q_{\pi_{\theta}}(s,a)$, and 2) the “actor” conducts the gradient ascent based on (8). The following iterative scheme is an illustrative actor-critic example:

1. 1)

   Given state $s$, take action $a\sim\pi_{\theta}(a|s)$, then observe the reward $r$ and next state $s^{\prime}$;

2. 2)

   (Critic) Update $Q$-function $Q_{\pi_{\theta}}(s,a)$ by TD learning;

3. 3)

   (Actor) Update policy parameter $\theta$ by

   $\displaystyle\theta\leftarrow\theta+\eta\,Q_{\pi_{\theta}}(s,a)\nabla_{\theta}\ln\pi_{\theta}(a|s);$

   (9)

4. 4)

   $s\leftarrow s^{\prime}$. Go to step 1) and repeat.

There are many actor-critic variants with different implementation manners, e.g., how $s$ is sampled from $\mu_{\theta}(s)$, how $Q$-function is updated, etc. See [2, Chapter 13] for more details.

We emphasize that the algorithms introduced above are far from complete. In the next subsections, we will introduce the state-of-the-art modern (D)RL techniques that are widely used in complex control tasks, especially for power system applications. At last, we close this subsection with the following two remarks on different RL settings.

This subsection presents the fundamentals of deep learning to set the stage for the introduction of DRL. Deep learning refers to the machine learning technique that models with multi-layer ANNs. The history of ANNs dates back to 1940s [39], and it has received tremendous interests in the recent decade due to the booming of data technology and computing power, which allows efficient training of wider and deeper ANNs. Essentially, an ANN is an universal parameterized mapping $y=\text{NN}(x;w)$ from the input features $x$ to the outputs $y$ with the parameters $w$. As illustrated in Fig. 3, an input feature vector $x$ is taken in by the input layer, then is processed through a series of hidden layers, and results in the output vector $y$. Each hidden layer consists of a number of neurons that are the activation functions, e.g. linear, ReLU, or sigmoid [40]. Based on a sample dataset $(x^{i},y^{i})_{i=1,\cdots,n}$, the parameter $w$ can be optimized via regression. A landmark in training ANNs is the discovery of the back-propagation method, which offers an efficient way to compute the gradient of the loss function over $w$ [41]. Nevertheless, it is pretty tricky to train large-scale ANNs in practice, and article [41] provides an overview of the optimization algorithms and theory for training ANNs. Three typical classes of ANNs with different architectures are introduced below. See book [42] for details.

1) Convolutional Neural Networks (CNNs) are in the architecture of feed-forward neural networks (as shown in Fig. 3) and specialize in pattern detection, which are powerful for image analysis and computer vision tasks. The convolutional hidden layers are the basis at the heart of a CNN. Each neuron $k=1,2,\cdots$ in a convolutional layer defines a small filter (or kernel) matrix $F_{k}$ of low dimension (e.g., $3\times 3$) and convolves with the input matrix $X$ of relatively high dimension, which leads to the output matrix $U_{k}=F_{k}\otimes X$. Here, $\otimes$ denotes the convolution operator⁶⁶6Specifically, the convolution operation is performed by sliding the filter matrix $F_{k}$ across the input matrix $X$ and computing the corresponding element-wise multiplications, so that each element in matrix $U_{k}$ is the sum of the element-wise multiplications between $F_{k}$ and the associated sub-matrix of $X$. See [42, Chapter 9] for a detailed definition of convolution., and the output $(U_{k})_{k=1,2,\cdots}$ is referred to as the feature map that is passed to the next layer. Besides, pooling layers are commonly used to reduce the dimension of the representation with the max or average pooling.

2) Recurrent Neural Networks (RNNs) specialize in processing long sequential inputs and tackling tasks with context spreading over time by leveraging a recurrent structure. Hence, RNNs achieve great success in the applications such as speech recognition and machine translation. RNNs process an input sequence one element at a time, and maintain in their hidden units a state vector $s$ that implicitly contains historical information about the past elements. Interestingly, the recurrence of RNNs is analogous to a dynamical system [42] and can be expressed as

where $x_{t}$ and $y_{t}$ are the input and output of the neural network at time step $t$, and $w:=(v,u)$ is the parameter for training. $s_{t}$ denotes the state stored in the hidden units at step $t$ and will be passed to the processing at step $t+1$. In this way, $s_{t}$ implicitly covers all historical input information $(x_{1},\cdots,x_{t})$. Among many variants, long-short term memory (LSTM) network [44] is a special type of RNNs that excels at handling long-term dependencies and outperforms conventional RNNs by using special memory cells and gates.

3) Autoencoders [45] are used to obtain a low-dimensional representation of high-dimensional inputs, which is similar to, but more general than, principal components analysis (PCA). As illustrated in Fig. 4, an autoencoder is in an hourglass-shape feed-forward network structure and consists of a encoder function $u=f(x)$ and a decoder function $y=g(u)$. In particular, an autoencoder is trained to learn an approximation function $y=\text{NN}(x;w)=g(f(x))\approx x$ with the loss function $L(x,g(f(x)))$ that penalizes the dissimilarity between the input $x$ and output $y$. The bottleneck layer has a much smaller amount of neurons, and thus it is forced to form a compressed representation of the input $x$.

For many practical problems, the state and action spaces are large or continuous, together with complex system dynamics. As a result, it is intractable for value-based RL to compute or store a gigantic $Q$-value table for all state-action pairs. To deal with this issue, function approximation methods are developed to approximate the $Q$-function with some parameterized function classes, such as linear function or polynomial function. As for policy-based RL, finding a capable policy class to achieve optimal control is also nontrivial in high-dimensional complex tasks. Driven by the advances of deep learning, DRL that leverages ANNs for function approximation or policy parameterization is becoming increasingly popular. Precisely, DRL can use ANNs to 1) approximate the $Q$-function with a $Q$-network $\hat{Q}_{w}(s,a):=\mathrm{NN}(s,a;w)$, and 2) parameterize the policy with the policy network $\pi_{\theta}(a|s):=\mathrm{NN}(a|s;\theta)$.

1) $Q$-Function Approximation. $Q$-network can be used to approximate the $Q$-function in TD learning (5) and $Q$-learning (6). For TD learning, the parameter $w$ is updated by

where the gradient $\nabla_{w}\hat{Q}_{w}(s_{t},a_{t})$ can be calculated efficiently using the back-propagation method. As for $Q$-learning, it is known that adopting a nonlinear function, such as an ANN, for approximation may cause instability and divergence issues in the training process. To this end, Deep $\bm{Q}$-Network (DQN) [47] is developed and greatly improves the training stability of $Q$-learning with the following two tricks:

$\bullet$ Experience Replay. Instead of performing on consecutive episodes, a widely used trick is to store all transition experiences $e:=(s,a,r,s^{\prime})$ in a database $\mathcal{D}$ called “replay buffer”. At each step, a batch of transition experiences is randomly sampled from the replay buffer $\mathcal{D}$ for $Q$-learning update. This can enhance the data efficiency by recycling previous experiences and reduce the variance of learning updates. More importantly, sampling uniformly from the replay buffer breaks the temporal correlations that jeopardize the training process, and thus improves the stability and convergence of $Q$-learning.

$\bullet$ Target Network. The other trick is the introduction of the target network $\hat{Q}_{\hat{w}}(s,a)$ with parameter $\hat{w}$, which is a clone of the $Q$-network $\hat{Q}_{w}(s,a)$. Its parameter $\hat{w}$ is kept frozen and is only updated periodically. Specifically, with a batch of transition experiences $(s_{i},a_{i},r_{i},s_{i}^{\prime})^{n}_{i=1}$ sampled from the replay buffer, the $Q$-network $\hat{Q}_{w}(s,a)$ is updated by solving

The optimization (12) can be viewed as finding an optimal $Q$-network $\hat{Q}_{w}(s,a)$ that approximately solves the Bellman Optimality Equation (4). The critical difference is that the target network $\hat{Q}_{\hat{w}}$ with parameter $\hat{w}$ instead of $\hat{Q}_{w}$ is used to compute the maximization over $a^{\prime}$ in (12). After a fixed number of updates above, the target network $\hat{Q}_{\hat{w}}(s,a)$ is renewed by replacing $\hat{w}$ with the latest learned $w$. This trick can mitigate the training instability as the short-term oscillations are circumvented. See [48] for more details.

In addition, there are several notable variants of DQN that further improve the performance, such as double DQN [49] and dueling DQN [50]. Particularly, double DQN is proposed to tackle the overestimation issue of the action values in DQN by learning two sets of $Q$-functions; one $Q$-function is used to select the action, and the other is used to determine its value. Dueling DQN proposes a dueling network architecture that separately estimates the state value function $V(s)$ and the state-dependent action advantage function $A(s,a)$, which are then combined to determine the $Q$-value. The main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying RL algorithm [50].

2) Policy Parameterization. Due to the powerful generalization capability, ANNs are widely used to parameterize control policies, especially when the state and action spaces are continuous. The resultant policy network $\mathrm{NN}(a|s;\theta)$ takes states as the input and outputs the probability of action selection. In actor-critic methods, it is common to adopt both the $Q$-network $\mathrm{NN}(s,a;w)$ and the policy network $\mathrm{NN}(a|s;\theta)$ simultaneously, where the “actor” updates $\theta$ according to (9) and the “critic” updates $w$ according to (11). The back-propagation method [41] can be applied to efficiently compute the gradient of ANNs.

When function approximation is adopted, the theoretical analysis on both value-based and policy-based RL methods is little and generally limited to the linear function approximation. Besides, one problem that hinders the use of value-based methods for large or continuous action space is the difficulty of performing the maximization step. For example, when deep ANNs are used to approximate the $Q$-function, it is not easy to solve $\max_{a^{\prime}}\hat{Q}_{w}(s,a^{\prime})$ for the optimal action $a^{\prime}$ due to the nonlinear and complex formulation of $\hat{Q}_{w}(s,a)$.

This subsection summarizes several state-of-the-art modern RL techniques that are widely used in complex tasks.

Frequency regulation (FR) is to maintain the power system frequency closely around its nominal value, e.g., 60 Hz in the U.S., through balancing power generation and load demand. Conventionally, three generation control mechanisms in a hierarchical structure are implemented at different timescales to achieve fast response and economic efficiency. In bulk power systems,⁷⁷7There are some other types of primary FR, e.g., using autonomous centralized control mechanisms, in small-scale power grids, such as microgrids. the primary FR generally operates locally to eliminate power imbalance at the timescale of a few seconds, e.g., using droop control, when the governor adjusts the mechanical power input to the generator around a setpoint and based on the local frequency deviation. The secondary FR, known as automatic generation control (AGC), adjusts the setpoints of governors to bring the frequency and tie-line power interchanges back to their nominal values, which is performed in a centralized manner within minutes. The tertiary FR, namely economic dispatch, reschedules the unit commitment and restores the secondary control reserves within tens of minutes to hours. See [56] for detailed explanations of the three-level FR architecture. In this subsection, we focus on the primary and secondary FR mechanisms, as the tertiary FR does not involve frequency dynamics and is corresponding to the power dispatch that will be discussed in Section III-C.

There are a number of recent works [57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71] leveraging model-free RL techniques for FR mechanism design, which are summarized in Table I. The main motivations for developing RL-based FR schemes are explained as follows.

• 1)

  Although the bulk transmission systems have relatively good models of power grids, there may not be accurate models or predictions on the large-scale renewable generation due to the inherent uncertainty and intermittency. As the penetration of renewable generation keeps growing, new challenges are posed to traditional FR schemes for maintaining nominal frequency in real-time.

• 2)

  FR in the distribution level, e.g., DER-based FR, load-side FR, etc., has attracted a great deal of recent studies. However, distribution grids may not have accurate system model information, and it is too complex to model vast heterogeneous DERs and load devices. In such situations, RL methods can be adopted to circumvent the requirement of system model information and directly learn control policies from available data.

• 3)

  With less inertia and fast power fluctuations introduced by inverter-based renewable energy resources, power systems become more and more dynamical and volatile. Conventional frequency controllers may not adapt well to the time-varying operational environment [68]. In addition, existing methods have difficulty coordinating large-scale systems at a fast time scale due to the communication and computation burdens, limiting the overall frequency regulation performance [58]. Hence, (multi-agent) DRL methods can be used to develop FR schemes to improve the adaptivity and optimality.

In the following, we take multi-area AGC as the paradigm to illustrate how to apply RL methods, as the mathematical models of AGC have been well established and widely used in the literature [56, 72, 73]. We will present the definitions of environment, state and action, the reward design, and the learning of control policies, and then discuss several key issues in RL-based FR. Note that the models presented below are examples for illustration, and there are other RL formulations and models for FR depending on the specific problem setting.

Voltage control aims to keep the voltage magnitudes across the power networks close to the nominal values or stay within an acceptable interval. Most recent works focus on voltage control in distribution systems and propose a variety of control mechanisms [74, 75, 76, 77, 78]. As the penetration of renewable generation, especially solar panels and wind turbines, deepens in distribution systems, the rapid fluctuations and significant uncertainties of renewable generation pose new challenges to the voltage control task. Meanwhile, unbalanced power flow, multi-phase device integration, and the lack of accurate network models further complicate the situation. To this end, a number of studies [79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96] propose using model-free RL for voltage control. We summarize the related work in Table II and present below how to solve the voltage control problem in the RL framework.

Energy management is an advanced application that utilizes information flow to manage power flow and maintain power balance in a reliable and efficient manner. To this end, energy management systems (EMSs) are developed for electric power control centers to monitor, control, and optimize the system operation. With the assistance of the supervisory control and data acquisition (SCADA) system, the EMS for transmission systems is technically mature. However, for many sub-regional power systems, such as medium/low-voltage distribution grids and microgrids, EMS is still under development due to the integration of various DER facilities and the lack of metering units. Moreover, an EMS family [100] with a hierarchical structure is necessitated to facilitate different levels of energy management, including grid-level EMS, EMS for coordinating a cluster of DERs, home EMS (HEMS), etc.

In practice, there are significant uncertainties in energy management, which result from unknown models and parameters of power networks and DER facilities, uncertain user behaviors and weather conditions, etc. Hence, many recent studies adopt (D)RL techniques to develop data-driven EMS. A summary of the related literature is provided in Table III. In the rest of this subsection, we first introduce the RL models of DERs and adjustable loads, then review the RL-based schemes for different levels of energy management problems.