MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learning

Deep Q-networks (DQN) [Mnih et al., 2015a] is the first successful applied off-policy deep RL algorithm. It has long been recognized that the overestimation in Q-learning could severely impair the performance [Thrun, 1993]. Double Q-learning [Hasselt, 2010, Van Hasselt et al., 2016] is proposed to solve overestimation issue for discrete action space. The overestimation issue still exists in continuous control algorithms such as Deterministic Policy Gradient (DPG) [Silver et al., 2014] and its variant Deep Deterministic Policy Gradient (DDPG) [Lillicrap et al., 2016]. [Fujimoto et al., 2018] introduced clipped double Q-learning (CDQ), which decreases the overestimation issue in DDPG. [Haarnoja et al., 2018] proposed Soft Actor Critic (SAC) algorithm which is based on maximum entropy RL framework [Ziebart, 2010] and combined with CDQ, resulting in a stronger algorithm. Maximum entropy RL framework encourages exploration capacity by adding policy entropy to optimization objectives. The CDQ approach introduces a slight underestimation issue [Lan et al., 2019, He and Hou, 2020a, b]. We apply the MEPG framework to DDPG and SAC algorithm, which achieves or surpasses the performance of compared methods even without introducing a technique to get a precise value function.

Ensemble methods use multiple learning algorithms to obtain better performance. They are also used in DRL [Zhang and Yao, 2019, Wiering and Van Hasselt, 2008, Osband et al., 2016, Chua et al., 2018] for different purposes. [Kurutach et al., 2018] showed that modeling error can be reduced by ensemble methods in model-based DRL. [Lan et al., 2019, Chen et al., 2020] tackled the estimation issue and gave unbiased estimation methods of the value function from the perspective of ensemble functions. [Agarwal et al., 2020] proposed Random Ensemble Mixture, which introduces a convex combination of multiple Q-networks to approximate the optimal Q-function. [Saphal et al., 2021] proposed a method for model training and selection in a single run. [Lee et al., 2021] proposed SUNRISE framework that uses an ensemble-based weighted Bellman backups and UCB exploration [Auer et al., 2002]. Unlike these methods, we only use a single model instead of multiple models to achieve our ensemble purpose.

It was recognized earlier that dropout [Srivastava et al., 2014] can improve the performance of DRL algorithms in video games [Lample and Chaplot, 2017, Vinyals et al., 2019b]. Because the dropout operator prevents complex co-adaptations of units in neural networks where a feature detector is only helpful in the context of several other feature detectors. Therefore, the effectiveness of the dropout in pixel-level input DRL is similar to the supervised learning tasks for image input. Our work extends the dropout method in high dimensional inputs settings to low dimension input scenarios, where the effect of preventing pixel-level features co-adaptations may not exist. [Kamienny et al., 2020] proposed a privileged information dropout method to improve sample efficiency and performance, but requires prior knowledge, i.e., an optimal representation for inputs. Our work does not require any auxiliary information. [Lütjens et al., 2019, Wu et al., 2021] obtain the neural networks model uncertainty with dropout through multiple forwards given the same inputs. The closest work to ours is DQN+dropout [Gal and Ghahramani, 2016] in a discrete environment, which also utilizes the uncertainty of Q-function induced by dropout. MEPG is different from [Gal and Ghahramani, 2016] for the following reasons. First, MEPG maintains consistency of the two sides of Bellman equation where [Gal and Ghahramani, 2016] does not. Failure to maintain consistency leads to a mismatch in the Bellman equation and then harms performance (see Table 2). Second, we use dropout for ensemble to improve performance and efficiency. [Gal and Ghahramani, 2016] uses dropout to obtain uncertainty thus using Thompson sampling to speed up convergence, without performance gain. Another approach to measuring uncertainty is to determine it using the variance generated by multiple Q-functions [Lee et al., 2021]. Our approach differs from previous works in that we neither use multiple Q-functions to estimate model uncertainty nor prevent pixel-level feature co-adaptation. We use the model ensemble property introduced by the dropout operator.

Computational resource consumption is a great challenge for the application of ensemble RL. Dropout operator has ensemble nature[Hara et al., 2016] and reduces computational resource consumption. Therefore, we consider the dropout operator to be applied to ensemble RL. Specifically, we consider integrating $2^{n}$ sub-models into a single model. We deploy the dropout operator [Srivastava et al., 2014] in our framework. The nature of the integration is guaranteed as follows. In the training phase, the neural network is equated to a sub-network after being acted upon by the sampled dropout operator. In the inference phase, the complete network that is not acted upon by the dropout operator is used, and the complete network is then equivalent to an ensemble network. Specifically, in the MEPG framework, the value network is trained under the above ensemble mechanism. When training the policy network, a complete value network (i.e., the ensemble value network) is used to induce the policy improvement phase. A feed-forward operation of a standard neural network for layer $l$ and hidden unit $i$ can be described as

where $f$ is an activation function, and $\mathbf{w}$ and $b$ represent the weights and biases respectively. We adopt the following dropout feed-forward style operation

where $p$ is the probability of an element to be set to zero and $\odot$ represents Hadamard product. The scale factor $\frac{1}{1-p}$ is added to ensure that the expected output from each unit would keep the same as the one without the dropout operator $\mathcal{D}_{\mathbf{m}}^{l}$. However, if the dropout operator directly acts on Bellman equation in this form, the algorithm cannot learn a precise Q-function due to the mismatch between the left and the right sides of Bellman equation (1). As a result, the algorithm fails to learn the value function, i.e., fail to estimate the policy $\pi$, failing the whole process. To tackle this issue, we introduce minimalist ensemble consistent Bellman update. Let $\mathcal{D}_{\mathbf{m}}^{l}$ be a dropout operator acting on layer $l$ with parameter $\mathbf{m}\sim\text{Bernoulli}(1-p)$. We define the form of minimalist ensemble consistent Bellman update as

which means we apply the same mask matrix $\mathbf{m}$ to both sides of Bellman equation. Thus minimalist ensemble consistent Bellman update can eliminate the mismatch without destroying the diversity of value functions so that good value functions are learned and then a good policy can be derived. The diversity and ensemble properties of value functions hold due to the dropout operator.

The MEPG framework is formulated by using the minimalist ensemble consistent Bellman update (equation (4)) in the policy evaluation phase and the conventional policy gradient methods in the policy improvement phase. At each learning step in MEPG framework, a $\mathcal{D}_{\mathbf{m}}^{l}$ operator is sampled, then policy evaluation phase proceeds with being acted upon by $\mathcal{D}_{\mathbf{m}}^{l}$. For policy improvement phase, the complete (i.e., ensemble) value function is used to train policy network. MEPG framework is summarized in Algorithm 1. We apply the MEPG framework to the DDPG algorithm and SAC algorithm, and call the resulting algorithms ME-DDPG and ME-SAC respectively.

In deterministic policy gradient style DRL algorithms, the policy can be updated by its value function [Silver et al., 2014, Lillicrap et al., 2016]

where the actor $\pi$ and critic $Q$ are parameterized by $\phi$ and $\theta$ respectively. The target network is updated by $\theta^{\prime}\leftarrow\eta\theta+(1-\eta)\theta^{\prime}$ at each time step, where $\eta$ is a small constant.

For the policy improvement phase in ME-DDPG, we utilize the original Deterministic Policy Gradient method (Equation (5)) without being acted by $\mathcal{D}_{\mathbf{m}}^{l}$ operator. And for the policy evaluation phase, the ME-DDPG algorithm utilizes equation (4). Besides, we take two tricks from TD3 algorithm [Fujimoto et al., 2018]: target policy smoothing regularization and delayed policy updates. ME-DDPG algorithm is summarized in Appendix 3.

The policy optimization objective of SAC [Haarnoja et al., 2018] is

ME-SAC keeps the original policy update style in SAC. The soft Q-function of ME-SAC can be obtained by minimizing the soft Bellman residual following equation 4,

The temperature parameter $\alpha$ can be given as a hyper-parameter or learned by minimizing

where $\mathcal{H}$ is a pre-given target entropy.

And for ME-SAC, we only use one critic with delayed policy updates. ME-SAC is summarized in Appendix 2. Note that any exploration technique can be coupled with MEPG (e.g. noise exploration in DDPG, entropy exploration in SAC).

We show that dropout operator acting on value networks can be seen as a deep GP. Therefore, MEPG keeps consistency of the dropout mask acting on both sides of Bellman equation, then the two deep GPs induced by the Dropout operator acting on both sides of Bellman equation are kept synchronized. This consistency improves the stability of Bellman equation compared to random mask situations.

Let $\hat{Q}^{\pi}$ be the output of action value function (a neural network), and a loss function $\mathcal{F}(\cdots)$. Let $\mathbf{W}_{i}$ be the weight matrix of $M_{i}\times M_{i-1}$ dimensions and the bias $\mathbf{b}_{i}$ of layer $i$ of dimension $M_{i}$ for $i\in\{1,2,\cdots,L\}$. We define Bellman backup operator as

which is different from Bellman backup defined in tabular setting [Sutton and Barto, 2018] due to importing auxiliary target network. Let $Q^{\pi}_{\text{True}}$ be the fixed point of Bellman equation (1) w.r.t. policy $\pi$. In the DRL setting, the critic network, which characterizes action value, is used to find the fixed point of Bellman equation w.r.t. current policy $\pi$ through multiple Bellman update. This optimization method is a different paradigm from supervised learning. For convenient, we denote the input and output sets for critic as $\mathcal{X}$ and $\mathcal{Q}$ where $\mathcal{X}\subseteq\mathcal{S}\times\mathcal{A}$. For input $x_{i}$, the output of the action value function is $\hat{Q}_{i}$. We only discuss policy evaluation problems, i.e., how to approximate the true Q-function given policy $\pi$, thus we omit the $\pi$ in the following statement. For a more detailed analysis on the dropout operator behind deep learning, we would recommend the readers check [Gal and Ghahramani, 2016] for more analysis on this topic. We often utilize modern optimization techniques like Adam [Kingma and Ba, 2015], which utilizes the $L_{2}$ regularization term in the learning process. Thus the objective for policy evaluation in conventional DRL can be formulated as

By minimizing Equation (10), we find the fixed point of Bellman equation, and then solve the policy evaluation problem. With the application of the dropout operator, the units of the neural network, are set to zero with probability $p$. Next we consider a deep Gaussian Process (GP) [Damianou and Lawrence, 2013]. Now we are given a covariance function

where $f$ is element-wise non-linear function. Equation (11) is a valid covariance function. Assume layer $i$ have a parameter matrix $\mathsf{W}_{i}$ with dimension $\mathsf{M}_{i}\times\mathsf{M}_{i-1}$ and we include all parameters in a set $\mathsf{\omega}=\{\mathsf{W}\}_{i=1}^{L}$. Let $p(\mathbf{w})$ in Equation (11) is the distribution of each row of $\mathsf{W}_{i}$. We assume that the dimension of the vector $\mathbf{m}_{i}$ for each GP layer is $\mathsf{M}_{i}$. Given some precision parameter $\varepsilon>0$, the predictive probability of the Deep GP model is

We utilize $q(\omega)$ to approximate the intractable posterior $p(\omega;\mathcal{X,Q})$. Note that $q(\omega)$ is a distribution over matrices whose columns are randomly set to zero. We define $q(\omega)$ as

where $\mathsf{G}_{i}$ is a matrix as variational parameters. The variable $z_{i,j}=0$ means unit $j$ in layer $i-1$ being zero as an input to layer $i$, which recovers the dropout operator in neural networks. To learn the distribution $q(\omega)$, we minimize the KL divergence between $q(\omega)$ and $p(\omega)$ of the full Deep GP

The first term in Equation (14) can be approximated by Monte Carlo method. We can approximate the second term in Equation (14), and obtain $\sum_{i=1}^{L}(\frac{p_{i}l^{2}}{2}\|\mathsf{G}_{i}\|_{2}^{2}+\frac{l^{2}}{2}\|\mathbf{m_{i}}\|_{2}^{2})$ with prior length scale $l$ (see section 4.2 in [Gal and Ghahramani, 2015]). Given the precision parameter $\varepsilon>0$, the objective of deep GP can be formulated as

We can recover Equation (10) by setting $\mathcal{F}(\mathcal{T}\hat{Q},\hat{Q})=-\log p(Q_{i}\mid x_{i};\hat{\omega})$. Note that the sampled $\hat{\omega}$ leads to the realization of the Bernoulli distribution $z_{i,j}$, which is equivalent to the binary variable $z_{i,j}$ in the dropout operator.

The above analysis shows that the policy evaluation process in the MEPG framework maintains two synchronized deep GPs, because both sides of Bellman equation are acted by the same dropout mask $\mathcal{D}_{\mathbf{m}}^{l}$. The uncertainty introduced by the dropout operator arises from the inherent property if the model and explicitly exists in the model. Thus the diversity and ensemble properties of value functions hold. Besides, this consistency naturally improves the stability of Bellman equation compared to random mask situations.

To evaluate the MEPG framework, we measure ME-DDPG and ME-SAC performance on PyBullet tasks compared with the SOTA model-free algorithm and recently proposed ensemble DRL algorithms such as ACE [Zhang and Yao, 2019], SUNRISE [Lee et al., 2021] and REDQ [Chen et al., 2020]. For TD3, ACE, SUNRISE and REDQ, we use the authors’ implementation with default hyper-parameters and keep the same hyper-parameters for DDPG and SAC. For a fair comparison, we replace one-time policy update every twenty times of critic optimization with one-time policy update every two-time of critic optimization in REDQ, namely REDQ(Ours). The empirical evaluation results show REDQ(Ours) is better than its original version. We run each task for one million time steps and perform one gradient step after each interaction step. Every $5,000$ time step, we execute an evaluation step over 10 episodes without any exploration operation in every algorithm. Our performance comparison results are presented in Table 1 and the learning curves are in Figure 2. The results show that most of the time our proposed algorithm ME-DDPG and ME-SAC are better than state-of-the-art methods without any auxiliary tasks. For the Pendulum family of environments, all algorithms are equally good. Our framework is best in terms of learning speed and final performance for the Ant, Walker2D, and Hopper environments while consuming fewer computational resources. More experimental results and details are in the Supplementary Material.

We conduct ablation studies to demonstrate the contribution of each individual component. We show the ablation results for ME-DDPG and ME-SAC in Table 2, where we compare the performance of removing or adding specific components from ME-DDPG or ME-SAC. Firstly, the effectiveness of the minimalist ensemble consistent Bellman update is investigated. We take the conventional Dropout operator acting on both sides of the Bellman equation, resulting in ME-DDPG-R and ME-SAC-R methods respectively. The empirical evaluations show that the proposed minimalist ensemble consistent Bellman update helps a lot and brings performance improvements. Secondly, we investigate three key components, i.e., dropout (DO), Target Policy Smoothing (TPS), and Delay Update (DU) in ME-DDPG. DO, DU and TPS help a lot. Thirdly, we perform a similar ablation analysis for the ME-SAC algorithm, where "FIX-ENT" means we adopt a fixed entropy coefficient $\alpha$. The ablation results of the ME-DDPG algorithm for DU, and DO are also applicable to the ME-SAC method. Here we find that automatic adjustment of entropy is extremely helpful. Additional experimental results and learning curves can be found in the Supplementary Material.

We evaluate the run time of one million time steps of training for each tested RL algorithm. In addition, we also quantify the number of parameters of neural networks corresponding to the evaluated algorithms in different environments. For a fair comparison, we keep the same update steps in REDQ(M). Note that REDQ(M) is fairly faster than the original REDQ (see section 5.1). And we cancel the evaluation process in this process. We show the run time test results in Table 3 and parameters quantification in Table 4. The MEPG framework consumes the shortest time with the same skeleton algorithms. Unsurprisingly, We find our algorithm ME-DDPG and ME-SAC act favorably compared to other algorithms we tested in terms of wall-clock training time. Our MEPG framework not only runs in one-third to one-seventh of the time of ensemble learning methods, but even less than the training time of model-free DRL algorithms. All run time experiments are conducted on a single GeForce GTX 1070 GPU and an Intel Core i7-8700K CPU at 2.4GHZ. The number of parameters of the MEPG framework is between 14% and 27% of the number of parameters of the tested ensemble algorithms, but our algorithms perform better than any tested one.

The MEPG framework only introduces one hyper-parameter $p$, which represents the probability of setting a neuron of the neural network to zero. We take eleven $p$ values in interval $[0,1]$. For visual simplicity, we normalize the data by $\text{Ret}^{\text{env}}_{p}=\text{Ret}^{\text{env}}_{p}/\max_{p}\{\text{Ret}^{\text{env}}_{p}\}$, where $\text{Ret}^{\text{env}}_{p}$ means the average of top five maximum average returns in a run over five trials of one million time steps for various p-value on env environment. We show the results in Figure 3. Each cell represents the result of a different p-value in various environments. The experimental results show that even large p-values, i.e., implicitly integrating enough models, do not cause the learning process to fail. Overall, the difference in performance between the different p-values is not very significant. For the Pendulum family of environments, the performance of the MEPG framework induced by various $p$ values is equally good. Relatively speaking, a smaller p-value gives better performance. Our MEPG framework is not extremely hyper-parameter sensitive. Therefore, we set $p=0.1$ as the default hyper-parameter setting for the ME-DDPG and ME-SAC algorithms.

We utilize the authors’ implementation of ACE, TD3, SUNRISE ,and REDQ without any modification as we discussed. For the implementation of SAC, we refer to [Yarats and Kostrikov, 2020]. And we do not change the default hyper-parameters for TD3, SAC, ACE, SUNRISE, and REDQ algorithms. For a fair comparison, we keep the same hyper-parameters to TD3 and SAC implementations respectively. But we only utilize one critic with its target. If the hyper-parameters of ME-SAC and ME-DDPG correspond to the SAC and DDPG, we use the same hyper-parameters. To implement minimalist ensemble consistent Bellman update, we take a $\mathbf{1}$ matrix, then let the $\mathbf{1}$ matrix pass through a dropout layer. The output of the dropout produces a consistent mask. We apply the same mask to the critic and its target. We give a detailed description of our hyper-parameters in Table 5.