Solving Deep Reinforcement Learning Benchmarks with Linear Policy Networks

Gradient-based deep reinforcement learning (DRL) has achieved remarkable success in various domains by enabling agents to learn complex behaviors in challenging environments based on their reward feedback, such as StarCraft Vinyals et al. (2019) and Go Silver et al. (2016). However, new methods are often benchmarked on simpler control tasks from OpenAI Gym, including the locomotion tasks from MuJoCo Haarnoja et al. (2018), or Atari games Mnih et al. (2015). While it simplifies the comparison between different approaches, these benchmarks may lack sufficient complexity, and performance may not always transfer to more complicated tasks. Additionally, several studies have indicated that DRL results are often hard to reproduce Islam et al. (2017), attributing these difficulties to the impact of the random seeds Henderson et al. (2018) and the choice of hyperparameters Eimer et al. (2023).

Evolution Strategies (ES) Rechenberg (1965); Bäck et al. (1991), a family of black-box optimization algorithms from the field of Evolutionary Algorithms (EAs) Bäck et al. (2023) have been studied as an alternative way to optimize neural network weights, as opposed to conventional gradient-based backpropagation Salimans et al. (2017); Such et al. (2017). An ES is used to learn a controller for an RL task by directly optimizing the neural network’s weights, which parameterize the RL policy. In this context, the ES is intrinsically an RL method that performs direct policy search, through neuroevolution Igel (2003). In supervised learning, gradient-based methods are often much more efficient than ES for training NN weights, though more likely to be trapped in local optima Mandischer (2002). For RL, the need to balance exploration with exploitation in gradient-based approaches incurs more training steps to learn an optimal policy Igel (2003), making ES an interesting alternative. While EAs are not necessarily more sample-efficient, ES can be more easily parallelized and scaled, offering the possibility for faster convergence in terms of wall-clock time, and, being a global search method, are less likely to get stuck in a local optimum Morse and Stanley (2016).

We benchmark three ES and three gradient-based RL methods on well-known RL tasks to understand the circumstances that favor ES over gradient-based methods. In particular, we study the optimization of policy networks that consist of a single linear layer, as they reduce the dimensionality of the problem Chrabąszcz et al. (2018); Rajeswaran et al. (2017). We compare these results to the larger networks that are used by common gradient-based methods. Our main contributions are as follows:

• •

  ES can find effective linear policies for many RL benchmark tasks. In contrast, methods based on gradient descent need vastly larger networks.

• •

  Contrary to the prevailing view that ES are limited to simpler tasks, they can address more complex challenges in MuJoCo. Gradient-based DRL only performs superiorly in the most challenging MuJoCo environments with more complex network architectures. This suggests that common RL benchmarks may be too simple or that conventional gradient-based solutions may be overly complicated.

• •

  Complex gradient-based approaches have dominated DRL. However, ES can be equally effective, are algorithmically simpler, allow smaller network architectures, and are thus easier to implement, understand, and tune (See Figure 1).

• •

  We find that advanced self-adaptation techniques in ES are often not required for (single-layer) neuroevolution.

The rest of the paper is organized as follows: Section 2 discusses the background and related work of ES and DRL, our algorithms are discussed in Section 3, the results are in Section 4, conclusions are in Section 5.

In RL, an agent learns from feedback through rewards and penalties from its environment Sutton and Barto (2018). RL problems are formulated as a Markov Decision Process $\langle S,A,P,R,\gamma\rangle$, where $S$ is the set of states in the environment, $A$ the set of actions available to the agent, $P$ the probability of subsequent state transitions, $R$ the reward function, and $\gamma\in[0,1]$ the discount factor  Bellman (1957). A policy $\pi$ can be computed of which action to take in each state. The policy in DRL is typically represented by a deep neural network to map states to actions (probabilities). RL aims to find the optimal policy $\pi^{*}$ that maximizes the expected cumulative reward of a state. RL algorithms approach this goal in different ways Plaat (2022). The most common techniques include value-function estimation Watkins and Dayan (1992); Mnih et al. (2015), policy gradient methods Williams (1992), actor-critic methods Konda and Tsitsiklis (1999); Schulman et al. (2017); Haarnoja et al. (2018), and learning a model of the environment Hafner et al. (2020); Plaat et al. (2023).

ES are a distinct class of evolutionary algorithms that are particularly suitable for optimization problems in continuous domains. ES begin with a population of randomly initialized candidate solutions in $\mathbb{R}^{n}$, with solutions represented as $n$-dimensional vectors denoted by $\mathbf{x}$ (like the policy) and a given objective function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ (like the reward). Via perturbations using a parameterized multivariate normal distribution, selection, and sometimes recombination, solutions evolve towards better regions in the search space Bäck et al. (1991). Evolving neural networks with EAs is called neuroevolution and can include the optimization of the network’s weights, topology, and hyperparameters Stanley et al. (2019). Using ES to evolve a neural network’s weights is similar to policy gradient methods in RL, where optimization applies to the policy’s parameter space.

Gradient-based deep RL has successfully tackled high-dimensional problems, such as playing video games with deep neural networks encompassing millions parameters Mnih et al. (2015); Vinyals et al. (2019). However, state-of-the-art ES variants are limited to smaller numbers of parameters due to the computational complexity of, for example, the adaptation of the covariance matrix of the search distribution. Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is often used for dimensionality lower than $n\leq 100$ Müller and Glasmachers (2018), and problems with a dimensionality $n\geq 10.000$ become nearly impossible due to the memory requirements Loshchilov (2014). However, recent advancements have restricted the covariance matrix, in its simplest form, to its diagonal, to reduce the computational complexity Ros and Hansen (2008); Loshchilov (2014); Nomura and Ono (2022). Others sample from lower-dimensional subspaces Maheswaranathan et al. (2019); Choromanski et al. (2019).

In 2015, DRL reached a milestone by achieving superhuman performance in Atari games using raw pixel input Mnih et al. (2015). This breakthrough marked a shift in RL towards more complicated, high-dimensional problems and a shift from tabular to deep, gradient-based methods. For simpler tasks, the CMA-ES has been used to evolve neural networks for pole-balancing tasks, benefiting from covariance matrix to find parameter dependencies, enabling faster optimization  Igel (2003); Heidrich-Meisner and Igel (2009). While the use of evolutionary methods for RL can be traced back to the early 90s  Whitley et al. (1993); Moriarty and Mikkulainen (1996), the paper by  Salimans et al. (2017) rekindled interest in ES from the field of RL as an alternative for gradient-based methods in more complicated tasks. Researchers showed that a natural evolution strategy (NES) can compete with deep RL in robot control in MuJoCo and Atari games due to its ability to scale over parallel workers. In contrast to deep methods where entire gradients are typically shared, the workers only communicate the fitness score and the random seed to generate the stochastic perturbations of the policy parameters. Studies have subsequently demonstrated that simpler methodologies can yield results comparable to NES, such as a classical ES Chrabąszcz et al. (2018) and Augmented Random Search Mania et al. (2018), which closely resembles a global search heuristic from the 1990s  Salomon (1998). In addition, when separating the computer vision task from the actual policy in playing Atari, the size of the neural network can be drastically decreased  Cuccu et al. (2018), and policies with a single linear layer, mapping states directly to actions, can effectively solve the continuous control tasks  Chrabąszcz et al. (2018); Rajeswaran et al. (2017). The development of dimension-lowering techniques, such as world models  Ha and Schmidhuber (2018); Hafner et al. (2020) and autoencoders  Hinton and Salakhutdinov (2006), also opens up new possibilities for ES to effectively solve more complex problems by simplifying them into more manageable forms.

We implement three ES methods and benchmark them against three popular gradient-based DRL methods.

In this section, we present the results of our experiments (more details, including training curves and tables with summary statistics, can be found in the Supplementary material). Here, we focus on a selection of environments (Figure 3).

In this study, we have studied different ways to optimize reinforcement learning policies with conventional deep learning gradient-based backpropagation methods as used in DQN, PPO, and SAC, and with three evolution strategy methods (ES). We have applied these methods to several classic reinforcement learning benchmarks. For these methods, we trained the regular deep network as conventionally used, as well as a neural network with no hidden layers, i.e., a linear mapping from states to actions, as a low-complexity controller for each environment. In many of the tested environments, the linear policies trained with the ES are on par or, in some cases, even better controllers than the deep policy networks trained with the gradient-based methods. Additionally, the gradient-based methods are often ineffective at training these simple policies, requiring much deeper networks. For our experiments on Atari, we find that by accessing the RAM memory of the Atari controller, ES methods can find a linear policy that is competitive with "superhuman" DQN Mnih et al. (2015). It should be noted that there are certain high-complexity environments where the deep gradient-based methods yield better policies, e.g. SAC for HalfCheetah. However, even for these environments, linear policies exist that are competitive and much more easily interpretable.

We conclude that conventional gradient-based methods might be overly complicated or that more complex benchmarks are required to properly evaluate algorithms. In fact, even for the most complex locomotion task included in our experiments, the Humanoid environment, the CMA-ES was able to find a linear policy that was competitive with state-of-the-art methods. As the ES are stochastic algorithms, they were not able to find these policies for every trial run, but our results show that such policies do exist. We expect the search landscapes for these environments to be deceptive and multimodal, and future work could help discover effective algorithms for more consistently training these linear policy networks, for example, using niching methods Shir and Bäck (2005). We hypothesize that gradient-based methods may struggle to find linear policies due to the multimodal nature of the search landscape, a phenomenon also seen in supervised learning Kawaguchi (2016). Counterintuitively, with gradient-based methods, it seems more straightforward to train deeper architectures than shallower ones with far fewer weights, as was also shown by Schwarzer et al. (2023). We note that gradient-based methods are essentially local search methods, requiring heuristically controlled exploration, while ES, in the early phases of optimization, are performing global search, producing more diverse solutions. This also becomes evident in the more simple environments, such as CartPole, where the ES are able to almost instantly sample the optimal policy, while the gradient-based methods have a much harder time.

Moreover, we find many counterexamples to the prevailing view that ES are less sample efficient than the gradient-based methods. For many of the low to medium-complexity environments, the ES are actually more sample efficient and require fewer environment interactions than the deep gradient-based methods. On the other hand, for the more complex environments, and with increasing dimensionality, we find that the ES can take more time steps to converge than the deep gradient-based methods. This is to be expected, as the self-adaptation mechanisms that are central to the ES become increasingly ineffective for larger dimensions Chrabąszcz et al. (2018); Müller and Glasmachers (2018). We have compared three ES, that, with increasing levels of complexity, adapt the shape of the mutation distribution in order to converge the search. Our results indicate that updates of the covariance matrix are often not required and that performing step size adaptation is sufficient. While we expect the search landscape to be multimodal, relative scaling and rotation of search coordinates seem absent, allowing isotropic mutations to be effective for these problems. This would also explain the effectiveness of the approach demonstrated in Maheswaranathan et al. (2019), which would be heavily impacted by conditioning on the search space. However, this may be explained because optimizing a single linear layer may exhibit less inherent variance and covariance than multiple layers.

Overall, we have demonstrated the potential of linear policies on popular RL benchmarks. We showed that ES are effective optimizers for these policies compared to gradient-based methods. Additionally, we note that ES are simpler in design, have fewer hyperparameters, and are trivially parallelizable. Hence, ES can perform more environment interactions within the same time frame Salimans et al. (2017). Moreover, evaluating linear policies is faster than evaluating one or sometimes several deep architectures, making the training much more expedient regarding wall-clock time. As the need for energy-efficient policy networks increases, our results warrant a closer look at ES for RL and training of simpler policies for tasks currently considered complex. For future work, we aim to extend our benchmark with more types of classical ES and strategies for multimodal optimization Preuss (2015). Additionally, it would be interesting to study the effect of the step-size adaptation methods in the presence of one or more hidden layers. Next, we will explore the potential of linear networks for other applications. Inspired by works such as Cuccu et al. (2018), we will look at more complex Atari games to see if they can also be solved by simple, energy-efficient means.