CGAR: Critic Guided Action Redistribution in Reinforcement Leaning

Under the classic setting of the off-policy actor critic learning procedure, at each interaction with the environment (environment step), the agent collects data from the environment under the policy network with the action distribution of ${\pi_{\phi_{a}(s)}}$. Then, given the updated data buffer $\mathcal{D}$, the critic is optimized to estimate the future reward. Then, the actor is updated to maximize the estimated expected future reward given the currently learned critic. The procedure forms a dependency circle and loops at each environment step during training.

At a certain environment step, suppose that the critic ${Q_{\phi_{c}}}$ is optimized to ${Q_{\phi_{c}^{i}}}$ after $i$ gradient step under the currently collected data $\mathcal{D}$. The loss function to optimize actor is generally written as,

When the loss function is minimized, the action with the highest probability predicted by the actor is the same as the action leading to the highest ${Q_{\phi_{c}^{i}}}$ predicted by the critic. Then the performance of the actor and the critic is the same. However, most of the time, the actor is less optimized to the optimal distribution of action given the current critic, which leads to poor performance compared with the current critic.

This learning procedure is similar to the knowledge distillation or teacher-student method [13]. In our situation, we use the Q value output by the critic to calculate the loss function to train the actor. Hence, critic and actor correspond to teacher and student, respectively. The performance gap between the student and teacher has been demonstrated in many previous works, and the gap exists even though the student network has the same size as the teacher network  [14, 15, 16]. In Section II-B, we also give an empirical demonstration of the motivation. As the performance of the critic is better than or at least equal to the actor in each environment step, we can expect that the critic can bring more expected discounted rewards than or at least equal to the actor during the RL training procedure. This paper proposes a CGAR algorithm to ameliorate the performance gap between the actor and critic.

We conduct experiments under supervised learning settings to empirically demonstrate the above motivation. In detail, We use MNIST [17] as a dataset and maintain two identical multilayer perceptron models, $M_{1}$ and $M_{2}$. We let $M_{1}$ fit the dataset and let $M_{2}$ fit $M_{1}$. Given input $x$ and its label $y$, the loss function of $M_{1}$ is: ${\mathcal{L}_{1}=CrossEntropy(M_{1}(x),y)}$, and the loss function of $M_{2}$ is: ${\mathcal{L}_{2}=CrossEntropy(M_{2}(x),M_{1}(x))}$. Under this setting, $M_{1}$ is the signal provided to $M_{2}$, and $y$ is the signal provided to $M_{1}$. Besides, $M_{1}$ and $M_{2}$ are updated one after another iteratively. This setting is quite similar to the off-policy actor critic algorithm in Section  II-A, where the reward is provided to calculate the critic’s target, and the Q is the signal provided to the actor. To simulate the loss function of critic, we further design another loss function, ${\mathcal{L}_{1}=MSE(M_{1}(x),\bar{y})}$, where $\bar{y}$ is the one-hot representation of $y$, and ${\mathcal{L}_{2}}$ remains unchanged. We test the evaluation accuracy of $M_{1}$ and $M_{2}$ every epoch over five seeds and report the results in Fig. 2. We can see that under both loss functions, $M_{2}$ learns more slowly than $M_{1}$, demonstrating our motivation’s correctness.

We propose our Critic Guided Action Redistribution (CGAR) based on the above motivation and empirical demonstration. In actor critic algorithms, given state ${s_{t}}$ at environment step $t$, the actor ${\pi_{\phi_{a}}}$ predicts the action distribution ${\mathcal{P}_{t}}$. After that, it samples action ${a_{t}}$ from ${\mathcal{P}_{t}}$ to interact with the environment.

In our algorithm, we first sample $K$ actions ${\{a_{t}^{0},a_{t}^{1},...,a_{t}^{K-1}\}}$ from $\mathcal{P}_{t}$. These actions construct the actions set ${\mathbb{L}_{a}^{t}}$.

Then we use the critic to predict the Q value for every action in ${\mathbb{L}_{a}^{t}}$. The Q value ${Q_{t}^{i}}$ is calculated on every action ${a_{t}^{i}}$ in ${\mathbb{L}_{a}^{t}}$ conditioned with state $s_{t}$. These Q values ${\{Q_{t}^{0},Q_{t}^{1},...,Q_{t}^{K-1}\}}$ construct the Q value set ${\mathbb{L}_{Q}^{t}}$.

We use the Q value ${Q_{t}^{i}}$ to calculate the probability ${p_{t}^{i}}$ of ${a_{t}^{i}}$. The probability set results from the Q value set calculated by the Softmax function. In this way, we could get the action probability distribution $\mathcal{P}_{t}^{\prime}$.

We use the Softmax function to make actions with large Q values more likely to be sampled while maintaining a certain degree of exploration. Finally, we select the action ${a_{t}}$ from the new distribution. Fig. 1 is our model’s diagram.

We apply CGAR to SAC. SAC is an off-policy algorithm based on the Maximum Entropy Principle. Its optimization goal is simultaneously maximizing both the expected return and the entropy. It learns a policy ${\pi_{\phi_{a}}}$, a Q value function ${Q_{\phi_{c}}}$, and a temperature coefficient ${\alpha}$, with parameters ${\phi_{a}}$, ${\phi_{c}=(\phi_{c}^{1},\phi_{c}^{2})}$, and ${\alpha}$ separately. The loss functions of SAC are defined below, which are introduced in the SAC paper.

Note that different from Eq. (1), there’s an entropy term in Eq. (9), which doesn’t affect the demonstration in Section II-A. Our complete algorithm is shown in Algorithm 1. Our unique operations are marked with red. And the deleted operations in SAC are marked with blue and strikethrough.

We implement CGAR on the SAC algorithm noted as CGAR-SAC. The implementation of the SAC comes from  [18]. We evaluate our method on OpenAI MuJoCo tasks, including tasks such as standing, walking, and running. The state of the agent consists of parameters such as positions and velocities. Action is a real-valued vector that represents the control of the agent’s joints. The purpose of model learning is to maximize the expected discounted rewards. We test the agent performance every 10,000 environment steps. We compute the mean episode returns an agent obtains over ten episodes for every evaluation. All results are over five different seeds, and we keep the minimum, maximum, and mean values over these seeds.

We compare the performance of CGAR-SAC with SAC. We counted the mean, maximum and minimum values of the average return obtained overall seeds during each evaluation and plotted them in Fig. 3. The curve represents the average return, and the shading represents the range between the maximum and minimum. From Fig. 3 we can see that in most tasks, CGAR-SAC converges faster and achieves better final performance than SAC, especially Cheetah Run, Walker Walk, and Finger Spin. And CGAR-SAC is not weaker than SAC in other tasks. We also counted the average return obtained overall seeds during each evaluation and calculated their mean over the entire training process. In Table I, we use the initials of the first letter of every environment as the table title, and the full names can be found in Fig. 3. The value of each item in the table represents the mean value of the average return, and the subscript denotes the standard deviation. From Table I, we can see that the mean value of the average return of our method during the training process is higher than SAC in every task, and the standard deviation is lower than SAC in most tasks. We can conclude that CGAR improves the sample efficiency of SAC.