Adapting to Reward Progressivity via
Spectral Reinforcement Learning

In decision making tasks involving compounding returns, such as stock market investing, it is common for the rewards received by the agent to increase in magnitude over time. An investor that starts out with little capital will achieve relatively small profits or losses early on, but if they are successful, their capital – and hence their potential profits and losses – will gradually increase. This property also arises in many games settings. For example, in the Atari game Video Pinball, the player can increase a “bonus multiplier” that increases the score paid per bumper hit. Since the bonus multiplier does not reset until the player dies, the rewards typically increase with time. In this paper, we refer to any task exhibiting this phenomenon as a progressive reward task.

We hypothesise that reward progressivity may be problematic for value-based deep reinforcement learning agents. Our rationale is that the temporal difference errors that arise under such algorithms typically scale with the magnitude of the training targets. Accordingly, experience collected from states with large expected returns may dominate training, causing the agent’s performance to degrade in other states. While this is rational in one sense – generally, it is more important to make accurate decisions in high-stakes situations – it is potentially harmful on progressive reward tasks. This is because, on progressive reward tasks, it may be necessary for the agent to perform well in relatively unrewarding regions before it can reach more rewarding regions. For example, in stock market investing, an investor can only reach states where they have large capital if they first perform well with small capital. Note too that the rewards need not be strictly progressive for this problem to arise; all that is needed is for the rewards to be progressive over some extended period.

Algorithms that clip the reward, such as DQN (Mnih et al., 2015), are less susceptible to this problem, because they do not perceive increases in reward magnitude beyond the clipping point. However, it is straightforward to construct examples where reward clipping masks the optimal solution. For example, in Bowling, clipping the rewards to [-1, 1] makes bowling a strike appear no better than hitting a single pin (Pohlen et al., 2018). While subsequent approaches make learning feasible without reward clipping (van Hasselt et al., 2016; Pohlen et al., 2018), we show in Section 3 that these methods are negatively impacted by strong reward progressivity. Motivated by this, we seek an algorithm that is capable of learning from unclipped rewards, while mitigating the negative impact that large rewards have on the agent’s ability to learn in less rewarding regions of the task.

To meet this aim, we propose Spectral DQN. Under this approach, rewards are decomposed into a multidimensional spectral reward, where each component is bound to [-1, 1]. The upper frequencies of the spectrum activate only when large magnitude rewards are received. For example, in Video Pinball, the lowest frequency activates on any score, while the upper frequencies only activate when the player has accumulated a large bonus multiplier. The bounded nature of the spectral rewards allows the agent to learn expected spectral returns without instability arising from large training targets. Meanwhile, the full, unclipped expected return can be recovered by summing across the return spectrum. (The full, technical definitions of these terms are provided in Section 4). In terms of addressing reward progressivity, a key advantage of this approach is that it allows us to balance the training loss and prevent the upper frequencies from receiving undue weight.

To test whether this approach helps mitigate the impact of reward progressivity, we perform two set of experiments: First, we apply Spectral DQN to two domains that we specifically designed to exhibit strong reward progressivity. While previous methods struggle on these tasks, failing to learn almost anything at all in the more extreme domain, Spectral DQN performs markedly better, making considerable progress on both tasks. Next, we apply our approach to a less constructed set of tasks; namely, 6 standard Atari games. Only some of these games exhibit reward progressivity, and none of them nearly so strongly as the extreme domains from earlier. While Spectral DQN does not outperform the benchmarks as noticeably in these domains (it is beaten by one of the benchmarks in a single game, but comfortably outperforms both benchmarks in three games), the results clearly show that Spectral DQN is not overfit to constructed domains. Moreover, since its outperformance in some of the games cannot be attributed solely to reward progressivity, it appears that the approach may offer additional advantages, as explained later in our analysis of the results.

In the Atari domain, where the DQN algorithm was first evaluated, the agent receives a reward equal to the score increase at each frame. However, since score magnitudes vary greatly across games, DQN clips all rewards to $[-1,+1]$. This constrains the size of the training targets so that it is easier to find a step size that performs well across the entire suite of games. While this heuristic turns out to perform well in Atari, a clear downside of the approach is that the agent becomes unable to distinguish large rewards from small rewards.

The Pop-Art algorithm (van Hasselt et al., 2016) offers a more principled solution. It trains from the unclipped reward, while normalising the training targets to have zero mean and unit variance. To ensure that the network’s predictions are preserved when the normalisation parameters are updated, the output of the final layer is scaled and shifted by an offsetting amount.

Pohlen et al. (2018) propose an alternative approach to handling unclipped rewards that reduces the variance of the training targets by applying a squashing function, $h$. The agent learns squashed action-values, $\tilde{Q}(s,a)=h(Q(s,a))$, via the transformed Bellman backup:

The authors prove that the transformed Bellman operator remains a contraction provided that both $h$ and $h^{-1}$ are Lipschitz continuous and $\gamma<1/L_{h}L_{h^{-1}}$ (where $L_{h}$ and $L_{h^{-1}}$ are the respective Lipschitz constants). In their experiments, they use the following squashing function:

where $\epsilon>0$ ensures that $h^{-1}$ is Lipschitz continuous.

For the remainder of this paper, we refer to this approach as target compression. In the Atari domain, Pohlen et al. found target compression to perform significantly better than Pop-Art. It has since become the dominant method for handling unclipped rewards in Atari, with several recent large-scale agents (Kapturowski et al., 2019; Badia et al., 2020a; b), including the state-of-the-art Agent57, favouring target compression over Pop-Art.

While both Pop-Art and target compression address reward variance across tasks, making it easier to devise a single learning configuration that performs well across a task suite, they were not designed to address intra-task reward variability. As such, we posit that they are ill-suited to addressing reward progressivity, since it is an intra-task property. To test this claim, we introduce two domains that are designed to exhibit extreme reward progressivity, and thus exacerbate the issue: Ponglantis and Exponential Pong.

Ponglantis. In Ponglantis, the game starts out exactly the same as the Atari game Pong, where valued-based deep reinforcement learning methods are known to perform well (Mnih et al., 2015). The reward scale in Pong is small relative to most Atari games; the player receives a +1 reward for hitting the ball past the computer paddle, and a -1 reward if they concede a point. In Ponglantis, however, if the player manages to score 10 points, the task immediately switches to Atlantis, where the reward scale is roughly 1000 times larger. The Atlantis phase ends upon the loss of a single life.

Exponential Pong. The game Exponential Pong is identical to regular Pong, except that the rewards are multiplied by $2^{n}$, where $n$ is the player’s current score.

We applied two agents to these games: Pop-Art and DQN+TC, where the latter is a version of DQN that we modified to use target compression and learn from the unclipped reward. The results of this experiment, averaged over 5 seeds, are shown in Figure 1. (For full experimental details, see the Appendix.) To simplify analysis of the Exponential Pong results, we have plotted the unexponentiated scores, i.e. the scores that the agents would have achieved under standard scoring in Pong. For context, the best possible score on standard Pong is +21, and the worst possible score is -21.

In Exponential Pong, which exhibits the more extreme reward progressivity (since the reward multiplier can potentially reach $2^{20}\approx$ 1 million), both agents essentially fail, providing strong support for our hypothesis that reward progressivity can be damaging, at least in extreme cases.

In the more moderate case of Ponglantis, Pop-Art fails to make meaningful progress beyond the low-scoring Pong phase of the task. The likely reason for this is that as soon as the agent starts reaching Atlantis and receiving large rewards, the target scaling mechanism employed by Pop-Art starts dividing the training targets by a large factor, heavily dampening the much smaller learning signals arising from the Pong phase of the task. On the other hand, DQN+TC is eventually able to make some progress into the Atlantis phase. The fact that DQN+TC outperforms Pop-Art is consistent with our hypothesis; while Pop-Art sees the reward scale jump by a factor of $\approx$1000 upon reaching Atlantis, the training targets of DQN+TC jump by a smaller factor, owing to the square root squashing function (Equation 2). In Appendix B.2, we perform some additional experiments in Ponglantis to find where the “cutoff points” for Pop-Art and DQN+TC lie, i.e. how big the jump in reward magnitude has to be for the agents’ performances to start deteriorating.

Given the above discussion, a natural question to ponder is whether DQN+TC might achieve better performance in Exponential Pong by employing a logarithmic squashing function instead of a square root. Unfortunately though, as Pohlen et al. (2018) point out to a reviewer of their work (see https://openreview.net/forum?id=BkfPnoActQ), a logarithmic transform does not have a Lipschitz continuous inverse, and thus the transformed Bellman is no longer guaranteed to be a contraction.

Based on the findings of the previous section, it seems that what we require is an algorithm where: (1) The training targets are constrained in magnitude, no matter how large the reward scale grows. (2) Regions of the task with small rewards yield meaningfully large temporal difference errors, even once much larger rewards have been discovered. Note that Pop-Art fails to meet the second criterion, while target compression fails to meet both, albeit it takes larger shifts in reward magnitude for the problems associated with reward progressivity to become apparent.

In this section, we introduce a reinforcement learning approach that meets both of the above criteria by leveraging the idea of spectral decomposition. This algorithm does not in and of itself address reward progressivity; in fact, we show that it behaves identically to the standard Q-learning algorithm (Watkins, 1989), given certain caveats. However, the properties of this algorithm facilitate adaption to progressive rewards, and in Section 5 we present a deep learning version of the approach that is capable of learning in domains with extreme reward progressivity, such as Exponential Pong.

In order to create a deep learning version of the Spectral Q-learning algorithm, we modify the DQN algorithm in an analogous way to which we modified the Q-learning algorithm in the previous section. We name the resultant algorithm Spectral DQN.

To parameterise all $N+1$ frequencies of the spectral action-value function via a single network, we multiply the number of outputs on the standard DQN network by the same count. One crucial detail here is that we initialise the weights and biases of the final layer to zero. This allows us to set $N$ to a conservatively large value, since it ensures that superfluous heads have no effect on training. Moreover, it ensures that the action-value calculations do not blow out due to the exponential weights in Equation 7 being applied to random noise.

For the training loss, we use a weighted sum of the temporal difference errors for each frequency:

where $w_{i}$ is the error weighting given to the $i$th frequency, $s$ is a state sampled uniformly from the replay memory, $a$ is the sampled action, $s^{\prime}$ is the next state, $r$ is the reward (with spectral components $r_{b}^{i}$ for $i\in\{0,\ldots,N\}$), $\theta$ are the parameters of the network being trained, and $\theta^{-}$ are the parameters of the periodically refreshed target network.

An important detail here is the approach we take to weighting the loss for each frequency. Given that the $ith$ frequency is weighted by $b^{i}$ when calculating the overall action-values, it is tempting to set $w_{i}=b^{i}$. However, this goes completely against one of the aims of our approach, which is to prevent the large reward frequencies from dominating the loss. In fact, we show in the next section that weighting by $b^{i}$ has a disastrous impact in the two extreme domains introduced earlier.

In light of the above, rather than weighting by $b^{i}$, we strive to give all frequencies equal weighting. However, we do not simply set $w_{i}=1$, because doing so would effectively give too much emphasis to the small reward frequencies. As pointed out by van Hasselt et al. (2016), the gradients propagated to the Q-network’s hidden layers depend quadratically on the scale of the targets (see Section 2.3 of their work). Since the lower frequencies have denser rewards, and hence larger training targets, they will dominate the gradients at the hidden layers if this effect is not accounted for. To address this, we set $w_{i}=1/\sigma_{i}^{2}$, where $\sigma_{i}$ is the standard deviation of the training targets for frequency $i$, calculated as a running average as in the Pop-Art algorithm. Finally, to ensure that this correction only affects the hidden layers, we unscale the gradients at the final layer to train as if $w_{i}=1$, as in standard DQN. (Implementing this is not especially difficult; the precise details can be found in our source code.) In Appendix B.1, we provide an ablation where we do not correct the hidden layer gradients, and we find that this has a significant adverse effect on performance.

To test how well Spectral DQN meets our aims, we first applied it to the two extreme domains from Section 3 – Ponglantis and Exponential Pong – where reward progressivity is clearly a key issue. In addition, to probe the importance of the frequency loss weighting, we included a variant of Spectral DQN with exponential loss weighting, i.e. $w_{i}=b^{i}$, which we argued against in the previous section. The results of this experiment, including the baseline results from Section 3, are shown in Figure 4, with each curve averaged over 5 seeds.

There are two clear findings here: (1) Spectral DQN is much better at handling extreme reward progressivity than the baselines. It scores much higher in Ponglantis, and makes far more meaningful progress in Exponential Pong, eventually learning to beat the computer player comfortably on average. (2) The loss weighting is of crucial importance, with the agent using exponential weights performing worse than even DQN+TC. The latter finding matches our expectations; as we noted in Section 3, target compression does at least somewhat mitigate the overemphasis given to large reward regions of the task, while Spectral DQN with $w_{i}=b^{i}$ gives heavy emphasis to such regions.

In Figure 4, we examine how the agents’ training errors evolve as they play Exponential Pong. The errors are broken down by the player’s current score $\in[0,20]$, with bluer lines corresponding to states with a low player score and redder lines corresponding to states with high player scores. To make the graphs comparable, we plot percentage TD errors, i.e.:

where the means are running averages, calculated from states sampled from the replay buffer. In all cases we report the true TD error, i.e. for DQN+TC we report the error on the uncompressed action-values, and for Spectral DQN we report the error on the full, summed action-values.

The instability of Pop-Art is quite apparent here: After a very brief period of stable learning, the TD errors start spiking as soon as the agent reaches even middling scores. Consistent with our earlier analysis, DQN+TC appears a lot more stable. For states with low player scores, the TD percentage errors hover at around 5%, although the agent never really learns accurate action-values for states where the player score exceeds $\approx$7 points. At first this might seem strange – since the reward scale increases exponentially with the player’s score, states with a high player score ought to be effectively prioritised by the loss function. However, there is an additional effect to consider here: As soon the agent’s predictions start to deteriorate for the lower-score states, its policy for these states likewise deteriorates and it starts struggling to reach the higher-score states. As such, the agent seems to get stuck in a middle ground, where it does not encounter the higher-score states often enough to learn accurate action-values there, nor for the estimates in the lower-score states to truly deteriorate. It appears that a TD percentage error of around 5% is a crucial cutoff line, where the agent is right on the cusp of being able to play well, but playing better, conversely, leads to deterioration.

In contrast to both Pop-Art and DQN+TC, when Spectral DQN reaches a new high score, it quickly learns accurate action-value estimates for the associated states, as indicated by the sharply dropping red lines in Figure 4. Moreover, the TD errors for the lower-score states remain stable as the agent improves, almost certainly because Spectral DQN’s loss does not give so much weight to large magnitude rewards. Finally, note that the agent’s sudden rise in performance at around $2.5\times 10^{7}$ frames in Figure 4 seems to coincide with the errors reducing below the 5% line, adding further credence to the analysis above.

To validate that Spectral DQN’s success in extreme domains does not come at the expense of performance in less constructed tasks, we also applied it to 6 standard Atari games. Moreover, we included 3 games where reward clipping is known to be detrimental, as per van Hasselt et al. (2016): Bowling, Centipede and Ms. Pacman. This was done to validate that the agent’s earlier success was not attributable to it somehow performing an implicit form of reward clipping. The results of these experiments are shown in Figure 5, with all curves averaged over 5 seeds.

While Spectral DQN’s performance relative to the baselines is not as exceptional here as in the extreme domains, it remains the strongest overall. Though the agent underperforms DQN+TC in Centipede, it noticeably outperforms both baselines in Bowling, Chopper Command and Video Pinball. On the other games, it performs at least on par with the benchmarks. Moreover, the fact that Spectral DQN’s performance compares favourably to Pop-Art in Bowling, Centipede and Ms. Pacman shows that it is truly responding to the unclipped reward.

Since Video Pinball exhibits reward progressivity, as explained in the introduction, Spectral DQN’s strong performance there fits well with our hypothesis. On the other hand, while Ms. Pacman also has progressive rewards — the player receives an increasing bonus for eating multiple ghosts with a single power pellet — these rewards come so close together that they are within the agent’s temporal horizon at the start of an episode. That is, there is not a small-expected-return phase preceding a large-expected-return phase, which we hypothesised to be a central component of the issue. (We return to this point and discuss further in the next section.)

Besides addressing reward progressivity, another possible advantage of Spectral DQN in high scoring games is simply that its training targets are more constrained. In games such as Video Pinball and Chopper Command, the expected returns can reach into the thousands, so even with target compression this may be problematic for DQN+TC. On the other hand, the only game where DQN+TC clearly outperforms Spectral DQN is Centipede, which is also high scoring. One possible explanation for this is that the reward structure in Centipede “confuses” Spectral DQN: Since large rewards trigger both low and high frequencies, it is likely that Spectral DQN learns a positive association between the various components of the return. However, in Centipede, losing a life triggers a stream of consecutive small rewards. Therefore, associating small and large rewards may cause Spectral DQN to predict an inflated return from dying.

There has been some conjecture that an advantage of combining distributional RL (Bellemare et al., 2017; Lyle et al., 2019) with function approximation is that it forces the agent to learn a richer representation. The same argument may also apply to our approach. In particular, in some tasks, small rewards are of more informational value than their face value represents. For example, in Video Pinball, where rewards range into the thousands, the play field includes “spinners” that pay only one point per hit. Despite their negligible face value, hitting the spinners can dramatically change the velocity of the ball, so giving the lower frequencies more weight in the training loss may promote a stronger representation. Similarly, in Bowling, certain frequencies only activate after bowling strikes and spares. Since the game does not have a strongly progressive reward structure, improved representation learning is a more plausible explanation of Spectral DQN’s outperformance.

In this paper, we brought attention to the idea of reward progressivity, a task property that poses a potential issue for value-based reinforcement learning agents. Specifically, an agent’s performance in relatively unrewarding regions of a task may degrade once it discovers larger rewards. To address this issue, we proposed Spectral DQN, which decomposes the expected return magnitudinally, allowing the losses arising across different regions of the task to be balanced. Our experiments showed this approach to be particularly effective in domains exhibiting extreme reward progressivity, while still yielding more than competitive performance in less constructed domains.

Our discussion of the Ms. Pacman results from the previous section raised an interesting point: There is clearly a link between reward progressivity and discounting. If it were somehow possible to train reliably with $\gamma=1$, a Ponglantis agent would be able to see that reaching the large, distant rewards in the Atlantis phase is contingent on first performing well in Pong. However, despite some recent progress, DQN-style agents typically destabilise with discounts very close to 1. Agent57 (Badia et al., 2020a) trains multiple value functions across a range of discounts, ranging as high as $\gamma=0.9999$. However, the authors note that $\gamma=0.9999$ generally yields very unstable learning, and they rely on a meta-controller to select smaller discounts in games where learning destabilises. As such, it does not offer a general solution to the problem. The idea of setting $\gamma=1$ also raises the issue of reward sparsity: If the agent is cognizant of large, distant rewards, then smaller, more immediate rewards may appear so negligible that the task effectively becomes a sparse reward problem.

We noted in Section 3 that another idea for addressing reward progressivity is to apply target compression, but with logarithmic squashing instead of a square root. However, as Pohlen et al. (2018) note, the inverse of a logarithmic mapping is not Lipschitz continuous, and thus the transformed Bellman is no longer guaranteed to be a contraction. On the other hand, one way to get around poor convergence in practice is to use a shaper discount factor. Van Seijen et al. (2019) show that it is possible to achieve performance comparable to DQN via a logarithmic squash, provided the discount is small ($\gamma=0.96$). However, they clip the rewards, which greatly constrains the gradient of the inverse mapping $h^{-1}$, and thus limits the degree to which their implementation transgresses the theory. Moreover, the performance of their method degrades significantly with $\gamma=0.99$, consistent with convergence issues (see Appendix C.2 in their work). Unfortunately, it is simple to provide examples where $\gamma=0.96$ will lead to poor performance. For example, in the game Bowling, there is a delay of about 100 frames between the ball being released and the subsequent reward. However, if the player bowls a spare, the reward is delayed by a further 100 frames. Thus, under a discount of 0.96, it is better to consistently bowl 9s than to bowl spares.

Our work fits into a broader line of research whereby reinforcement learning agents seek to predict quantities other than just the mean expected return (Jaderberg et al., 2016; Bellemare et al., 2017; Fedus et al., 2019). Of this related work, one of the most similar approaches to ours is that of Romoff et al. (2019), who learn the value function in a modular way by decomposing based on the discount factor. Distributional reinforcement learning (Bellemare et al., 2017; Dabney et al., 2018b; a) is another closely related approach. Finally, an older line of work in this vein is decomposed RL (Russell & Zimdars, 2003; Sprague & Ballard, 2003; Van Seijen et al., 2017), whereby an agent is responsible for predicting only a portion of the action-value function. While some of these works bear many surface-level similarities to our own — the distributional C51 agent (Bellemare et al., 2017) in particular bears quite a similar network architecture to ours — on a deeper level, they are not addressing the same problem. Leveraging a probabilistic loss, as in C51, could foreseeably help to address the large training errors that arise in tasks like Exponential Pong. However, the method assumes that returns are constrained to [-10, 10], and it is unclear how this could be extended to unclipped rewards without requiring an intractable number of atoms in the distribution, or greatly reducing the precision of the atoms.