Reinforcement learning with experience replay
and adaptation of action dispersion

We consider an actor-critic algorithm with experience replay. The algorithm keeps a window of length $M$ of previous events and continuously optimizes the policy. We postulate that the dispersion of the distribution of the current policy should be sufficient to produce actions that will enable the evaluation and selection of future policies, i.e., these actions should be likely in future policies. However, the future policies are unknown. We assume that they will be as different from the current policy as the current policy is different from those that produced the actions registered in the current memory buffer.

Following the above postulate, we propose two rules to optimize $\theta_{\eta}$:

R1:

For the $i$-th event registered in the memory buffer, the mode of the distribution that has produced the action $a_{i}$ should be likely according to the current policy and the state $s_{i}$.

R2:

For the same event, the action $a_{i}$ should be likely according to the current policy and the state $s_{i}$.

The above rules (illustrated in Fig. 1) are intended to have the following effects: 1) When the action distribution for a given state, $s$, changes due to learning, the current action distribution for the state $s$ is dispersed, thereby enabling to evaluate a broad range of behaviors expected to be exercised in the course of the learning. 2) However, when the policy approaches the optimum and the pace of its change decreases, the distribution becomes less dispersed, enabling more precise action choices. Eventually, the action distribution converges to a deterministic choice when the policy converges.

The mode of the distribution that has produced the action $a_{i}$ is defined as

for $\theta_{\mu}$ and $\theta_{\eta}$ used at time $i$. Following the aforementioned rules R1 and R2, to optimize $\theta_{\eta}$ we minimize the loss

averaged over the events collected in the memory buffer. $\alpha>0$ is a coefficient.

The algorithm presented here is Actor-Critic with Experience Replay and Adaptive eXploration, ACERAX. It is based on the Actor-Critic structure shown in Section 3, experience replay, and $n$-step returns. The algorithm uses a critic, $\bar{V}(s;\nu)$, an approximator of the value function with weights $\nu$.

At each time $t$ of the agent-environment interaction, the following tuple is registered

As the agent-environment interaction continues, previous experience is being replayed; that is, Algorithm 1 is being recurrently executed.

In Line 1, the algorithm selects an experienced event to replay. In Line 2, it determines the relative quality of $a_{i}$, namely the temporal difference multiplied by a softly truncated density ratio. $\theta$ is changing due to learning. Thus the conditional distribution $(a_{i}|s_{i})$ is now different than it was at the time when the action $a_{i}$ was executed. The product of density ratios in $e$ accounts for this discrepancy in distributions. To limit the variance of the density ratios, the soft-truncating function $\psi_{b}$ is applied, e.g.,

for $b>1$. In the ACER algorithm (Wawrzyński,, 2009), the hard truncation function, $\min\{\cdot,b\}$ is used for the same purpose, which is limiting density ratios necessary in designating updates due to action distribution discrepancies. However, soft-truncating distinguishes the magnitude of density ratio and may be expected to work slightly better than the hard truncation.

In Line 3, an improvement direction of the parameters of critic, $\nu$, is computed. $\Delta\nu$ is designed to make $\bar{V}(\cdot;\nu)$ approximate the value function better.

In Line 4, an improvement direction for the actor parameter $\theta_{\mu}$ is computed. The increment $\Delta\theta_{\mu}$ is designed to increase/decrease the likelihood of occurrence of the sequence of actions $a_{i}$ proportionally to $e$. The $p$ function is a penalty for improper values of $\bar{\mu}(s_{i};\theta_{\mu})$, e.g., exceeding a box to which the actions should belong.

In Line 5, an improvement direction for the actor parameter $\theta_{\eta}$ is computed, for $l_{i}(\theta_{\eta})$ defined in (8).

The improvement directions $\Delta\nu$, $\Delta\theta_{\mu}$ and $\Delta\theta_{\eta}$ are applied in Line 6 to update $\nu$, $\theta_{\mu}$, and $\theta_{\eta}$, respectively, with the use of either ADAM, SGD, or another method of stochastic optimization. They may be applied in minibatches, several at a time.

Each learning run lasted for 3 million timesteps. Every 30000 timesteps, the training was paused, and a simulation was made with frozen weights and without exploration for 5 test episodes. An average sum of rewards within a test episode was registered. Each run was repeated five times.

We have taken implementation SAC and PPO from (Raffin et al.,, 2021), and the hyperparameters that assure the state-of-the-art performance of these algorithms from (Raffin and Stulp,, 2020). For each environment, hyperparameters for ACER and ACERAX, such as step-sizes, were optimized to yield the highest ultimate average rewards. The values of these hyperparameters are reported in the appendix.

For each environment, we tried $\alpha=0,0.001,0.01,0.1,1$. The results are depicted in Fig. 3. It is seen that the learning is not very sensitive to this parameter. $\alpha=0$ means that the previous modes of action distributions and the previous actions are made likely according to the current policy. $\alpha>0$ means that the previous actions are made likely, which is intended to be a form of regularization: It prevents the policy from degrading to a deterministic one too early. The experiments suggest that performance is almost insensitive to this parameter when $\alpha\in[0,1]$. We choose $\alpha=0.1$ for the rest of the experiments.

We have optimized in a grid search constant standard deviations for actions for ACER in these environments. In all environments we selected $\sigma=0.4\approx\exp(-1.2)$. In another experiment, we verify the average outputs of the $\bar{\eta}$ network depending on its initialization. To this end, we impose different initial biases in the output neurons of this network. Afterward, we register the outputs of this network over time. Fig. 4 presents the trajectories of average $\min_{i}\bar{\eta}_{i}(s_{t};\theta_{\eta})$ and $\max_{i}\bar{\eta}_{i}(s_{t};\theta_{\eta})$. It is seen that regardless of the initialization, these outputs converge quite close to $-1.2$. We also see that $\min_{i}\bar{\eta}_{i}(s_{t};\theta_{\eta})<\max_{i}\bar{\eta}_{i}(s_{t};\theta_{\eta})$ which justifies the need of designating standard deviations of different action dimensions separately.

Fig. 5 presents learning curves for all four environments for SAC, PPO, ACER, and our proposed ACERAX. It is seen that all algorithms exhibit a similar speed of learning and ultimate rewards, with ACERAX performing very well according to both these criteria. (In fact, ACERAX yields the best average ultimate performance in 3 out of 4 environments, but this result is not statistically significant.) However, the reference algorithms control exploration-exploitation balance with heuristic or problem-dependent coefficients. In ACERAX, this balance is based on an an universal principle that is somehow affected by the $\alpha$ coefficient but is not very sensitive to this coefficient.

Exploration is necessary for reinforcement learning, but it inevitably deteriorates currently expected rewards. Best previous approaches to tuning the amount of exploration online are based on additional rewards for action distribution entropy. But the scale of these rewards is generally problem-dependent, as this is never known in advance what fraction of the original rewards need be traded for action distribution entropy to assure proper exploration.

The approach analyzed here is based on the assumption that the policy is optimized on the data in the replay buffer. The current action distribution dispersion should be sufficient to support policy evaluation and selection when this action will be replayed from the buffer in the future.

Our approach generally yielded good performance in the experiments. That happened at the cost of an additional neural network, $\bar{\eta}$, that controlled the action distribution dispersion. Our experiments suggest that this network should be much smaller than the $\bar{\mu}$ network – it had ten times fewer neurons in each layer. We interpret this latter condition as follows: It should be impossible for the $\bar{\eta}$ network to overfit to the experience, as it would result in nearly zero action distribution dispersion for some states.