Action Advising with Advice Imitation
in Deep Reinforcement Learning

Deep reinforcement learning (RL) has made it possible to build end-to-end learning agents without having to handcraft task-specific features, as it is showcased in various challenging domains such as StarCraft II Vinyals et al. (2017) and DotA II Berner et al. (2019) in the recent years. These feats make deep RL a great candidate to be employed in complex real-world sequential decision-making problems. However, achieving the reported levels of performance usually requires millions of environment interactions due to the deep learning induced complexity as well as the exploration challenges in RL itself. Even though this may seem negligible in most of the experimental domains considering the immense amount of computing power available to be utilised through parallel simulations, it usually poses a problem in the real-world scenarios due to the interaction costs and safety concerns. Furthermore, since RL is an inherently online learning approach, it is desired for the agents to be continually learning after they have been deployed too. For these reasons, it is crucial to improve sample efficiency in deep RL, which is actively investigated in several lines of research. One promising approach to tackle this setback is leveraging some legacy knowledge acquired from other entities such as agents, programs or humans.

Peer-to-peer knowledge transfer in deep RL has been investigated in various forms to this date da Silva et al. (2020b). A popular approach, namely Learning from Demonstrations (LfD), focuses on incorporating a previously recorded dataset in the learning process. By taking some dataset generated by another competent Hester et al. (2018) or imperfect Gao et al. (2018) peer, the learning agent tries to make the most out of the available information through off-policy learning and extra loss terms. Another promising, yet under-investigated class of techniques, namely Action Advising Torrey and Taylor (2013), aims to take advantage of a competent peer interactively when there is no pre-recorded data. The learning agent acquires advice in the form of actions from a teacher for a limited number of times defined by a budget that resembles the practical limitations of communication and attention. This approach is especially beneficial in the situations where there is no way to access the actual task before the online training, data collection is costly or the relevant data that will do the most contribution in the learning can not be determined. Action advising methods in deep RL today are quite limited and therefore have several shortcomings. An important one of these as we address in this study is not being able to make further use of the advice beyond its collection.

The scope of the action advising problem is generally limited to answering “when to ask for advice?”. It is commonly not of any interest how the collected advice is utilised by the student agent’s task-level RL algorithm, e.g., how it is stored, replayed, discarded; especially since these are dealt with by the studies that focus on off-policy experience replay dynamics in general (Schaul et al., 2016; De Bruin et al., 2015), or the specific case of having demonstration data as in LfD. However, even without interfering with the student’s task-level learning mechanism, it is still possible to make more of advice through reuse. Current action advising algorithms in deep RL have no way of telling if they have asked for advice in a very similar or even identical state already in the learning session. Thus, they do not record these in any way, and usually end up requesting advice from the teacher redundantly. In order to address this, we incorporate a separate neural network to do behavioural cloning (Pomerleau, 1991) on the samples (state-action pairs which are equal to the state-advice pairs in the context of action advising) collected from the teacher. This network then will be able to serve as a state-conditional generative model that will let us sample advice for any given observation. However, since this model should also have a notion of distinguishing the recorded states from the unrecorded ones to avoid producing false advice for unfamiliar states, we also propose incorporating a well known regularisation mechanism called Dropout (Srivastava et al., 2014) within this network to serve as an epistemic uncertainty estimator (Gal and Ghahramani, 2016) which will allow the student to determine whether the state is recorded by comparing this estimation with a threshold.

Our contributions in this study are as follows: First, we show that it is possible to generalise teacher advice across similar states in deep RL with high accuracy. Second, we present a RL algorithm-agnostic approach to memorise and imitate the collected advice that is suitable with the deep RL settings. Finally, we demonstrate that advice reuse via imitation provides significant boosts in the learning performance in deep RL even when it is paired with a simple baseline like early advising.

The majority of action advising studies to date have been conducted in classical RL settings. Torrey and Taylor (2013) was the first study to formalise action advising within a budget constrained teacher-student framework. Specifically, they studied the teacher-initiated scenario and came up with several heuristics to distribute the advising budget to maximise the student’s learning performance, such as early advising and importance advising. This work was then extended to introduce several new state importance metrics Taylor et al. (2014). In Zimmer et al. (2014), action advising problem was approached as a meta-RL problem itself. Instead of relying on heuristics, the authors attempted to learn the optimal way to distribute advising budget by using a measurement of the student’s learning acceleration as the meta-level reward. Besides these studies that only consider the teacher’s perspective, Amir et al. (2016) explored student-initiated and jointly-initiated variants considering the impracticality of requiring the teacher’s attention constantly. They achieved results on-par with the previous work without requiring the teacher full-time. Zhan et al. (2016) shed light in the theoretical aspects of action advising problem by using a more general setting involving multiple teachers and demonstrated the effects of having good or bad teachers. In da Silva et al. (2017), the authors adopted the teacher-student framework in cooperative multi-agent RL where the agents learn from scratch and hold no assumptions of their teacher roles and expertise. By proposing state counting as a new heuristic in this setting, they successfully accelerate team-wide learning of independent learners. More recently, learning to teach concepts was further investigated in Fachantidis et al. (2019) with a focus on the properties that make for a good teacher. In this work, besides learning when to advise, the teachers also learn what to advise. Similarly, Omidshafiei et al. (2019) adopted the meta RL approach, this time as a deep RL scale. They considered a team of two agents that learn to cooperate from scratch in tabular multi-agent tasks. Zhu et al. (2020) is one of the most recent studies conducted in tabular settings. The idea of reusing the previously collected advice in order to make the most out of a given small budget was studied. By devising several heuristics to serve as reusing schedules, they demonstrated promising results that outperform the algorithms incapable of advice reusing.

The domain of deep RL is a fairly new area for action advising where the primary choice is the student-initiated approaches. Chen et al. (2018) is one of the first studies to explore the idea of action advising in deep RL. They combined the LfD paradigm Hester et al. (2018) with interactive advice exchange under the name of active learning from demonstrations to collect demonstration data on-the-fly to be utilised via imitation capable loss terms as used in Hester et al. (2018). Furthermore, they proposed using epistemic uncertainty estimations of the student agent’s model to time this advice. Later, Kim et al. (2020) was proposed as an extension of Omidshafiei et al. (2019). This time, meta deep RL to address learning to teach idea was applied in the problems that are deep RL in the task-level. Through multiple centralised learning sessions, agents in a set of cooperative multi-agent tasks were made to learn taking student and teacher roles as needed in order to improve team-wide knowledge. To do so, they adopted hierarchical reinforcement learning (Nachum et al., 2018) to deal with the meta-level credit assignment problem of the teacher actions. In Ilhan et al. (2019), the formal action advising framework was scaled up to deep RL level for the first time. Similarly to da Silva et al. (2017), a team of agents in a cooperative multi-agent scenario were made to exchange advice by embracing teacher or student roles as needed. This was accomplished by using random network distillation (RND) Burda et al. (2018) to replace state counting with state novelty, hence introducing a new heuristic that is applicable in non-linear function approximation domain. Later on, da Silva et al. (2020a) proposed the idea of uncertainty-based action advising as in Chen et al. (2018), though without employing any additional loss terms. To access uncertainty estimations, they studied the case of student agent with a multi-headed network architecture in particular. In a more recent work Ilhan and Pérez-Liébana (2020), student-initiated scenario is further studied to devise a more robust heuristic able to handle extended periods of absence of teacher as well as having no requirements in the student’s task-level architecture by completely decoupling the module that is responsible for advice timing from the student’s model. Even though this method also uses the state novelty heuristic proposed in Ilhan et al. (2019), they operated on the advised states directly rather than every encountered state.

Clearly, none of the related work in deep RL addressed further utilisation of collected advice, besides Chen et al. (2018) which does it through interfering with the student’s learning mechanism (via a custom loss function), unlike our approach. The study that is closest to the idea we present in this paper is Zhu et al. (2020); though, it is limited to the tabular RL domains only. Such a setting makes it more straightforward for the agent to precisely memorise the state-advice pairs in a look-up table to be able to reuse anytime. Furthermore, the executed advice usually has an instantaneous impact on the agent behaviour in the case of tabular RL, which presents unique options to assess their usefulness. Since these advantages are absent in deep RL, our work deals with different challenges than those in Zhu et al. (2020).

We follow the standard MDP formalisation given in Section 3.1 in our problem definition. In this setting, a student agent that employs an off-policy deep RL algorithm performs learning in an episodic single-agent environment through trial-and-error interactions. It receives an observation $s_{t}$ and then executes an action $a_{t}$ generated by its policy $\pi_{S}$ to receive a reward $r_{t}$ at each timestep $t$, in order to maximise its cumulative discounted rewards in any episode. According to the teacher-student paradigm (Section 3.5) we adopt, there is also an isolated peer that is competent in this same task, and is referred to as the teacher. For a limited number of times defined by the action advising budget $b$, the student is allowed to acquire an action advice from the teacher for the particular state $s$ it is in. While the teacher can have its own teaching strategies to generate actions to advise, in our setting, we determine the action to be advised greedily from the teacher’s behaviour policy as $\pi_{T}(s)$. This is a commonly followed approach with the assumption of the teacher and the student’s optimal task-level strategies are equivalent. The student considers this advice as a part of a high-reward strategy and follows them upon collection. In this final form of the problem, the student’s objective is to spend its budget at the most appropriate times to maximise its learning performance.

We aim to devise a method that will enable the student to memorise the collected advice to be able to re-execute them in the similar states; therefore, avoiding wasting its budget in redundant states and potentially being able to follow the teacher advice many more times than its budget. In tabular RL, this is trivial to achieve simply by storing the advised actions paired with the states in a look-up table. When it comes to deep RL where any particular observation is not expected to be encountered more than once, however, there needs to be a generalisable approach. For this purpose, we propose the student agent to employ a separate behavioural cloning module, which consists of a neural network as the state-conditional generative model $G_{\omega}\colon\mathcal{S}\rightarrow\mathcal{A}$. By training $G_{\omega}$ in a supervised fashion with the obtained state-advice pairs (stored in a buffer $D$) to minimise the negative log-likelihood loss $\mathcal{L}(\omega)=\sum_{(s,a)\in D}-logG_{\omega}(a\mid s)$, the student can imitate the teacher’s advice to reuse them accordingly. However, this method does not have any mechanisms to prevent the student from generating incorrect advice from the states it has not collected. Therefore, we also employ Dropout regularisation in $G_{\omega}$ in order to grant this behavioural cloning module a notion of epistemic uncertainty through measuring the variance in the outputs obtained from multiple forward passes for a particular input state. We denote this uncertainty estimation by $G_{\omega}^{\mu}(s)$. The states $G_{\omega}$ is trained on will be less susceptible to the variance caused by the dropout and yield smaller uncertainty values. By this means, the student can determine how likely a state is to be already recorded as advised when it comes to reusing them, and can make a decision according to a threshold.

An obvious question regarding the feasibility of reusing advice in deep RL arises here: can the teacher’s advice be generalised over similar states accurately? As we investigate in the experiments in Section 7, actions generated by the teacher policy usually span over similar states. Clearly, the uncertainty threshold to consider a state as recorded is responsible for the trade-off between the reusing amount and the accuracy of the self-generated teacher advice. A small threshold value makes the student reuse its budget in fewer states with higher accuracy, whereas a larger value results in more frequent reusing with lower accuracy.

The detailed breakdown of our approach is summarised with an emphasis on the proposed modifications as follows (as also shown in Algorithm 1): The student starts with a randomly initialised $G_{\omega}$ and empty $D$. At each timestep $t$ with the (observed) state $s_{t}$ and an undecided action $a_{t}$, the student first checks if $D$ has any new samples. As soon as $D$ reaches the size defined by $n_{D}$, $G_{\omega}$ is trained with mini-batch gradient descent over the samples in $D$ for $k_{bc}$ iterations. Afterwards, if the environment was reset (a new episode started), the student determines whether to enable advice reuse via imitation for this particular episode with a probability of $\varepsilon_{reuse}$, which is combined with other conditions too later on in the algorithm. The idea behind employing this condition is to ensure that the student can also execute its own exploration policy in order to increase the data diversity in its replay memory, which is crucial to improve the quality of learning. Furthermore, determining this variable on an episodic basis lets the agent follow consistent policies in the exploration steps, rather than dithering between two policies. In the next phase, the student deals with the advice collection. We adopt the simple yet strong baseline of early advising here. According to this, the agent just collects advice without any conditions until its budget runs out. In the next phase, the student decides whether to reuse advice generated by its $G_{\omega}$. There are several conditions to be satisfied for this to occur in addition to the advice reuse being allowed for this particular episode. Firstly, $a_{t}$ must be non-determined, which implies the agent has not collected any advice from the teacher already. Secondly, $G_{\omega}$ must be already trained, so that it can generate meaningful actions. Then, the student also checks if its own action $\pi_{S}(a\mid s_{t})$ is explorative. This condition limits the action advising actions to the exploration steps only in order to prevent overriding the student’s actual policy which may result in lack of Q-value corrections and cause deteriorative effects when too much advising occurs. Finally, it is checked whether $G_{\omega}^{\mu}(s_{t})$ is smaller than the reuse threshold $\tau_{reuse}$. Incorporating such threshold is important to limit the imitated advice to the states that have low uncertainty according to $G_{\omega}$ to achieve higher accuracy of generating correct teacher actions. On one hand, having this threshold too high would make the student consistently follow $G_{\omega}$ which would result in a dataset with lower diversity. On the other hand, if $\tau_{reuse}$ is set too small, then $G_{\omega}$ would be ignored in the most of the cases and the student would be following its own exploration policy. After all these steps, if $a_{t}$ is still non-determined, the student follows its own policy and decide $a_{t}$ by $\pi_{S}(a\mid s_{t})$.

In order to have a significant complexity level as well as the challenges that are relevant to the deep RL methods in our experiments, we chose three Atari 2600 games from the commonly used Arcade Learning Environment (ALE) Bellemare et al. (2013) as our evaluation domain:

• •

  Enduro: The player controls a racing car in a long-distance track over multiple in-game days. In each day, if the player manages to pass a certain number of other cars ($200$ in the first day, $300$ in the rest) in the race, it gets to advance to the next day. Progression during the days is visualised by different colour schemes that resemble the day-night cycle. Furthermore, there are other factors of seasonal events that affect the gameplay such as fogs and icy patches appearing on the road. Finally, as the days progress, the game increases in difficulty due to the other cars’ behaviour becoming more aggressive.

• •

  Freeway: In this game, the objective is to cross a chicken across a highway comprised of ten lanes with vehicles traversing in different directions and speeds. If the player hits the cars along the way, it gets pushed back towards starting point. Every time the player manages to reach the goal, it acquires a reward and gets teleported back at the starting point.

• •

  Pong: This game consists of two paddles on the each side of the screen and a ball traversing around. The paddles are controlled by one player each. The players must hit the incoming balls to avoid them passing through their side as well as getting them thrown back at the opponent. If a player lets the ball pass through the gap behind its paddle, the opponent earns $1$ point. In the single-agent variant of this game used in our study, the player controls the right side paddle while the other one is controlled by a built-in AI.

Each of these games has an observation size of $160\times 210\times 3$, representing RGB images of the game screen that are produced at $60$ frames per second (FPS). To make experimenting in these games computationally tractable, we employ some preprocessing steps that are also followed commonly in other studies Castro et al. (2018). First of all, each observation is made greyscale and resized down to the size of $80\times 80\times 1$. Since the games run at a high FPS, the frame that is shown to the player is set to be only every $4$th one (which is composed of the maximum pixel values of previous $3$ frames), and the player’s actions are repeated for the skipped frames. Moreover, since these games contain a fair amount of partial observability, such as the direction of the ball in Pong, the final form of the observation to be perceived by the player is made to be a stack of $4$ pre-processed frames with a size of $84\times 84\times 4$ (which contains the information of the most recent $16$ actual game frames). In order to deal with the varying range of reward scales and reward mechanisms within these games, every reward obtained in a single step in the game is clipped to be in $[-1,1]$. Finally, every game episode is limited to last for maximum $108$k frames, which corresponds to approximately $30$ minutes of actual gameplay time in real-life.

Another set of modifications also take place to introduce more stochasticity within the games to turn them into more challenging RL tasks. In the beginning of the games, the player takes no-op actions for a random number of times in $[0,30]$, to simulate the effect of having different initial states. Additionally, with a probability of $0.25$, the actions executed by the player are repeated for an additional step, which is referred to as sticky actions.

The goal of our experiments¹¹1Code for our experiments can be found at https://github.com/ercumentilhan/naive-advice-imitation is to demonstrate that it is possible to generalise the teacher advice to the unseen yet similar states with our method, and that it is an effective way of improving performance of action advising, in complex domains especially. Therefore we choose the games described in Section 5 as our test-beds. The set of the student agent variants we compare are listed as follows:

• •

  No Advising (None): No action advising procedure is followed; the student learns as normal.

• •

  Early Advising (EA): The student follows early advising heuristic to distribute its advising budget. Specifically, the teacher is queried for an advice at every step until the budget runs out.

• •

  Early Advising with Advice Reuse via Imitation (AR): The student follows our proposed strategy (Section 4) combined with early advising heuristic. It starts off by greedily asking for advice until its budget runs out; then, it activates its behavioural cloning module to imitate and reuse the previously collected advice in the remaining exploration steps.

All student agent variants employ the identical task-level RL algorithm which is DQN with double Q-learning and dueling networks enhancements, and $\epsilon$-greedy policy as the exploration strategy. The convolutional neural network structure within the DQN in input-to-output order is as follows: $32$ $8\times 8$ filters with a stride of $4$, $64$ $4\times 4$ filters with a stride of $2$, $64$ $3\times 3$ filters with a stride of $1$, followed by a fully-connected layer with $512$ hidden units and multiple streams that add up in the end (dueling). Additionally, the student agent variant AR also incorporates a behaviour cloning module, which is a neural network with an identical structure minus the dueling stream. All the layer activations are set to be ReLU. The hyperparameters are tuned prior to experiments and kept the same across all experiments can be seen in Table 1.

In this teacher-student setup, we also need a teacher from which the student can get good quality action advice. For this purpose, we trained a DQN agent for each of these games for $10$M steps ($40$M actual game frames) to achieve a competent level performance in each.

The experiments are conducted by executing every student variant through a learning session $3$M steps ($12$M actual game frames) for every game. The learning steps are kept relatively small compared to the teacher training since it is expected for the students to achieve high performance much quicker with the aid of advice. Through the learning sessions, the agents are also evaluated at every $25$k^th step in a separate instance of the environment for $10$ episodes. During evaluation, any form of exploration and teaching is disabled in order to assess the actual proficiency of the students.

In terms of action advising setup, we set the action advising budget as $10$k steps which corresponds to only approximately $0.3\%$ of the interactions in a learning session and also to almost one third of a full game episode ($27$k steps). Besides the budget, our proposed method AR also uses some additional hyperparameters which were tuned prior to the full length experiments and are kept the same across every game. The dataset size $n_{D}$ to train $G_{\omega}$ is set as $10$k which is the action advising budget as we employ early advising prior to behavioural cloning training. The number iterations to train $G_{\omega}$ is set as $50$k. Episodic advice reuse probability $\varepsilon_{reuse}$ is set as $0.5$ meaning that the student will follow $G_{\omega}$ in half the episodes (in the appropriate states). Finally, advice reuse uncertainty threshold $\tau_{reuse}$ is set as $0.01$ (determined empirically) and kept the same across all games. In the experiments with AR, we also record the actual advice actions generated by the teacher at every step (not seen by the student) to have access to the ground-truth values to measure the accuracy of the behavioural cloning module. Every particular experiment case is repeated and aggregated over $3$ different random seeds.

The results of our experiments are presented in Figures 2,2 and Table 2. Figure 2 contains the plots for the evaluation scores obtained by None, EA and AR modes of the student in the games of Enduro, Freeway, Pong. In Figure 2, the plots of the advice reuse trends of AR in this set of games are displayed as cumulatively (top row) and in every $100$ steps windows (bottom row). These plots are limited to the first $500$k steps to only consider the exploration stage determined by the agent’s $\epsilon$-greedy schedule. Purple lines here represent all advice reuses combined, while the green lines indicate only the correctly imitated (in terms of being equal to the ground-truth teacher advice) advice pieces. These results are also reported in Table 2 in the numerical form where the evaluation scores are broken down in two parts of final value and area-under-the-curve, which represent the final agent performance and the learning speed, respectively. Furthermore, the table also contains the total number of exploration steps taken, as well as the percentage of the number of reused advice in the exploration steps and the percentage of correctly imitated advice in total number of reused advice (denoted in parentheses).

In the evaluation scores, we see different outcomes in each of these games. In Enduro, we see that AR provides a significant amount of jump start and performs the best in terms of learning speed while being far ahead of EA and None which are quite similar. When it comes to the final performance however, while EA and AR both outperform None, they do not differ much from each other. In Freeway, EA and AR perform very similarly in terms of learning speed and final performance with AR being slightly ahead of EA. However, they outperform None significantly. This shows that it matters to be advised initially, though their repetitions may not always yield much acceleration in learning. Finally, in Pong, we see a great difference between the performances in every aspect. Our AR comes out far ahead than its closest follower EA both in terms of final score and learning speed. This is an example of how getting a very little advice in the beginning as well as repeating them across further explorative actions can cause a great impact on the learning. Overall, AR manages to be the best in every game and suffers no performance loss even with high advice utilisation (as high as  $104$k in Freeway) which was shown to be harmful to learning in previous studies. Even though its performance boost over None seems to be not huge in every scenario, it should be noted that this is the case of it being combined with EA baseline. With more complicated methods, AR can be capable of training its imitation learning module with a more diverse set of experience and therefore, have a larger coverage which can potentially yield superior performance.

The task-level performance of our approach is affected primarily by two factors: the accuracy of advice imitation and its coverage/usage in the remainder of the exploration steps (the process of reusing). Therefore, we also analyse the advice reuse statistics of AR to form links between these outcomes. First of all, it should be noted that the decreasing trend in these plots is caused by the $\epsilon$-greedy annealing. Enduro is the game with the smallest advice reuse rate as well as the lowest imitation accuracy. This is possibly because of the game episodes lasting long regardless of the agent’s performance, which is likely to reduce the proportion of the familiar states according to the behavioural cloner. In Freeway, we observe a fairly high advice reuse rate with high accuracy of imitation. However, this is not reflected in the performance difference obtained versus EA, unlike in Enduro and Pong. Finally, in Pong, where the performance improvement is the most significant, advice reuse ratio seems to be similar to Enduro, but with far higher imitation accuracy.

Clearly, as we see from all these results combined, we can say that it is definitely a viable idea to extend the teacher advice over future states through imitation since this can be achieved with relatively high accuracy. However, even when we have access to these imitated competent policies, it is still non-trivial to construct a good exploration policy. While a higher advice reuse rate produces a more consistent exploration policy with less random dithering, it also has the risk of limiting the sample diversity in the replay memory, which can be problematic especially if the imitation quality is also poor. As long as the reuse amount does not get excessively high, it is safe to have the imitation learning accuracy around these reported levels, which makes tuning the uncertainty threshold straightforward. This is especially important for the realistic applications where it is not possible to access the tasks to tune such hyperparameters beforehand.

Finally, we also analyse our approach’s computational burden, which may be the primary concern when adopting it. Specifically, it involves two extra operations: behavioural cloning network training and uncertainty estimations. The former happens only once in the beginning and therefore is negligible. The uncertainty estimations that require multiple forward passes (which is $100$ in our experiments) happens in every exploration step and was found to cause a maximum of $2\times$ slowdown in our experiments. Considering that the exploration steps only spans approximately $10\%$ of a learning session, we can expect the runs to be taking at most $10\%$ longer in total when AR is employed in a similar setting to ours; and, this becomes even smaller when the learning sessions last longer in terms of the total number of environment steps. Clearly, this is a small setback considering the sample efficiency benefits our method brings.

In this study, we developed an approach for the student to imitate and reuse advice previously collected from the teacher. This is the first time such an approach has been proposed in deep reinforcement learning (RL). In order to do so, we followed an idea similar to behavioural cloning, employing a separate neural network that is trained with the advised state-action pairs via supervised learning. Thus, this module can imitate the teacher’s policy in a generalisable way that lets us apply it to the unseen states. We also incorporated a notion of epistemic uncertainty via dropout in this neural network to be able to limit the imitations to the states that are similar to the advice collected states.

The results of the experiments in $3$ Atari games have shown that it is a feasible idea to accurately generalise a small set of teacher advice over unseen yet similar states in future. Furthermore, our approach of employing behavioural cloning was found to be a successful way of achieving this, as it yielded a considerably high accuracy of imitation in multiple games. Additionally, reusing these self-generated advice across the exploration steps provided significant improvements in the learning speeds and the final performances without any over-advising induced performance deterioration. Therefore, our method can be considered as a promising enhancement to the existing action advising methods, especially since it is also very straightforward to implement and tune, with only a small computational burden. Finally, it was also seen that utilisation of such imitated advice policies to construct good quality exploration is non-trivial and requires further investigation.

Our study lies at the intersection of action advising and exploration in RL and can be extended in various interesting ways. It is unclear how far the different qualities of imitation and reuse rates can affect performance in one particular game; it will be a worthwhile study to analyse these. Furthermore, evaluating the advice in terms of its contribution to learning progress is a promising direction to take.