Learning on a Budget via Teacher Imitation

Deep Reinforcement Learning (RL) has been proven to be a successful approach to solve decision-making problems in a variety of difficult domains such as video games [1], board games [2] or robot manipulation [3]. However, achieving the reported performances is not entirely straightforward. One of the most critical setbacks in deep RL is the exhaustive training processes that usually require many interactions with the environment. This occurs mainly due to the RL inherent exploration challenges as well as the complexity of the incorporated function approximators, e.g. deep neural networks. To this date, there has been a remarkable amount of research effort to overcome the sample inefficiency by devising advanced exploration strategies [4]. In addition to these, other lines of work that focus on leveraging some legacy knowledge to tackle these issues have also been studied extensively with great success.

The ability to learn by utilising the prior experience instead of starting from scratch is an essential component of intelligence. In RL, this idea has been investigated in various forms that are tailored for different problem settings. Imitation Learning (IL) [5] studies the concept of mimicking an expert behaviour presented in a pre-recorded dataset without allowing any RL rewards from the environment itself. Similarly, Learning from Demonstrations (LfD) [6][7] extends this idea to incorporate the RL interactions and rewards to further surpass the experts. Some other approaches such as Policy Reuse [8] study the ways of speeding up the agent’s learning by leveraging the past policies directly, instead of datasets.

In this paper, we study a different problem setup where it is not possible to have any pre-generated datasets or to directly obtain the useful policies themselves; instead, the learning agent only has access to some peer(s) over a limited communication channel. Such a setting is especially relevant for the scenarios with unknown task specifications (prior to learning and deployment) and non-transferable yet beneficial policies, e.g. humans, non-stationary agents, in the loop. A flexible framework that is tailored for this setting is called Action Advising [9]. According to this, agents exchange knowledge between each other in the form of actions to speed-up their learning progressions. However, the number of these peer-to-peer interactions are limited with a budget to resemble the real-world limitations, which essentially converts the problem into determining the best possible way to utilise the available budget. Based on the peer that is in charge of driving these interactions, action advising algorithms can either be student-initiated, teacher-initiated or jointly-initiated.

Action advising algorithms in deep RL have obtained promising results with an emphasis on student-initiated strategies [10][11][12]. While the majority of these focus on addressing when to ask for advice question, some recent approaches [13] have also investigated the ways to further utilise the collected advice by imitating and reusing the teacher policy. Despite these developments, there are currently several significant shortcomings present. These techniques often employ some threshold hyperparameters to control the decisions to initiate advice exchange interactions which play a key role in their efficiencies. However, these parameters are sensitive to the learning state of the models as well as the domain properties. Therefore, they need to be tuned very carefully prior to execution, which involve unrealistically accessing the target tasks for trial runs. Furthermore, the studies to further leverage the teacher advice beyond collection are currently in their early stages and do not provide a complete solution to the problem besides addressing the advice reusing aspect.

In this paper, we present an all-in-one student-initiated approach that is capable of collecting and reusing advice in a budget-efficient manner, by extending [13] in multiple ways. First, we propose a method for automatically determining the threshold parameters responsible for the decisions to request and reuse advice. This greatly alleviates the burden of task-specific hyperparameter tuning procedures. Secondly, we follow a decaying advice reuse schedule that is not tied to the student’s exploration strategy. Finally, instead of using the imitated policy only for reusing advice as in [13], we incorporate this policy to determine and collect more diverse advice to construct a more universal imitation policy.

The rest of this paper is structured as follows: Section II outlines the most relevant previous work. In Section III, the background knowledge that is required to understand the paper is provided. Afterwards, we describe our approach in detail in Section IV. Then, Section V presents our evaluation domain and experimental procedure. In Section VI, we share and analyse the experiment results. Finally, Section VII wraps up this study with conclusions and future remarks.

Action advising techniques with budget constraints were originally invented and studied extensively in tabular domains. In [9], the teacher-student learning procedure was formalised for the first time and some solutions from the teacher’s perspective were proposed. This was later extended with some new heuristics [14] as well as with a meta-learning approach [15]. Later on, the action advising interactions are also studied in student-initiated and jointly-initiated forms [16] which were also adopted in multi-agent problems where the agents take student and teacher roles interchangeably [17]. In a recent work [18], the idea of reusing the previously collected advice is investigated to improve the learning performance and also to make a more efficient use of the available budget.

Deep RL is a considerably new domain for the action advising studies. [19] introduced a novel LfD setup in which the demonstrations dataset is built interactively as in action advising. To do so, they employed uncertainty estimation capable models with LfD loss terms integrated in the learning stage. [10] extended the jointly-initiated action advising [17] to be applicable in multi-agent deep RL for the first time. This study replaced the state counters that were used to assess uncertainty in the tabular version [17] with state novelty measurements with the aid of Random Network Distillation (RND) [20]. In [11], an uncertainty-based advice collection strategy was proposed. According to this, the student adopts a multi-headed neural network architecture to access epistemic uncertainty estimations as in [19]. Later, [12] further studied the state novelty-based idea of [10] to devise a better student-initiated advice collection method. Specifically, they made RND module to be updated only for the states that are involved in advice collection. That way, the student was ensured to benefit from the teacher regardless of its own knowledge about the states to tackle the special cases of belated inclusion of the teacher. More recently, [13] studied the advice reuse idea in deep RL. In this work, an imitation model of the teacher is constructed partially with the collected advice. Furthermore, Dropout regularisation [21] is also incorporated in this model to enable it to make uncertainty-aware decisions when it comes to reusing the self-generated advice.

Among these studies in deep RL, only [19] and [13] investigated the concept of advice reuse. Our paper differs from them in several ways. The idea in [19] require using uncertainty estimation capable models in the student’s RL algorithm. Moreover, the RL algorithm also needs to be modified to have the LfD loss terms. In contrast, the algorithm we propose does not require the student to have any specific RL models or loss functions. Thus, the student agent can be treated as a blackbox, which lets our method to be applied to a wider range of agent types. [13] performs advice reuse as a separate module, as we described here. However, they rely on the previously proposed advice collection strategies instead of taking advantage of its own imitation module to manage the advice collection process via uncertainty. Furthermore, some of the hyperparameters in [13] limit its practical applicability due to being difficult to tune, which we also address in this work.

We adopt the MDP formalisation presented in Section III in our problem definition. The setup in this study includes an off-policy deep RL agent (student) with policy $\pi_{S}$ learning to perform some task $T$ in an environment with continuous state space and discrete actions. There is also another agent with policy $\pi_{T}$ (teacher) that is competent in $T$. The teacher is isolated from the environment itself, but is reachable by the student via a communication channel for a limited number of times defined by the advising budget $b$. By using this mechanism, the student can request action advice $a_{T}=\pi_{T}(s)$ for its current state $s$. The objective of the student in this problem is to maximise its learning performance in $T$ by timing these interactions to make the most efficient use of $\pi_{T}$.

Our approach provides a unified solution for addressing when to ask for advice and how to leverage the advice questions. In addition to the RL algorithm, the student is equipped with a neural network $G_{\omega}$ with weights $\omega$ that are not shared with the RL model in any way. The student also has a transitions buffer $D$ with no capacity limit that holds the collected state-advice pairs. By using the samples in $D$, $G_{\omega}$ is trained periodically to provide the student an up-to-date imitation model of $\pi_{T}$ to make it possible to reuse the previously provided advice. Moreover, $G_{\omega}$ is also used to determine what advice to collect by being regarded as a representation of $D$’s contents. Obviously, making these decisions require $G_{\omega}$ to have a form of awareness of what it is trained on (in terms of samples). Therefore, $G_{\omega}$ employs Dropout regularisation in the fully-connected layers to have an estimation of epistemic uncertainty denoted by $G_{\omega}^{\mu}(s)$ for any state $s$ as it is done in [21][13]. None of these components share anything with or require access to the student’s RL algorithm. This is especially advantageous when it comes to pairing up our approach with different RL methods.

At the beginning of the student’s learning process, $\omega$ is initialised randomly and $D=\emptyset$. Then, at every timestep $t$ in state $s_{t}$, the student goes through $3$ stages of our algorithm: Collection, Imitation, Reuse. The remainder of this section describes these stages with the line number references to the complete flow of our algorithm summarised in Algorithm 1.

The collection stage (lines 13-19) remains active from the beginning until the student runs out of its advising budget $b$. At this step, the student attempts to collect advice if its current state has not been advised before. This is determined by the value of $G_{\omega}^{\mu}(s_{t})$. If it is higher than the uncertainty threshold $\tau$ (which is set automatically in the imitation stage), it is decided that $s_{t}$ is has not been advised before; thus, the student proceeds with requesting advice. However, if $\tau$ is undetermined, this request is carried out without performing any uncertainty check.

The imitation module is responsible for training $G_{\omega}$ and tuning $\tau$ accordingly. This stage (lines 20-22) is always active, but it is only triggered when these conditions that are checked at every timestep $t$ are met: the student has collected $n_{min}$ new samples in $D$ (since the last imitation) or the student has taken $t_{min}$ steps (since the last imitation) with at least $n_{min}/2$ new samples in $D$. Here, $n_{min}$ and $t_{min}$ are hyperparameters. These are set in order to keep the number of imitation processes within a reasonable number while also ensuring $G_{\omega}$ remains up-to-date with the collected advice. On one hand, if $G_{\omega}$ was updated for every new state-advice pair, it would be a very accurate model of $D$’s contents, but the total training times would be a significant computational burden. On the other hand, if $G_{\omega}$ was updated infrequently, it would not cause any computational setbacks; however, it would not be a good representation of the collected advice either.

Once the imitation is triggered, $G_{\omega}$ is trained for $k_{init}$ iterations (if it is the first ever training; else, for $k_{periodic}$ iterations) with the minibatches of samples drawn randomly from $D$. This process resembles the simplest form of behavioural cloning where the supervised negative log-likelihood loss $\mathcal{L}(\omega)=\sum_{(s,a)\in D}-logG_{\omega}(a\mid s)$ is minimised. Afterwards, $\tau$ is updated automatically to be compatible with the new state of this imitation network. This is done by measuring $G_{\omega}^{\mu}$ and storing them in a set $U$ for each $s_{i}$ in $D$ that satisfies $a_{i}=\operatorname*{argmax}_{a}G_{\omega}(a\mid s_{i})$. Then, the uncertainty value that corresponds to the $p^{th}$ percentile (hyperparameter) in the ascending-order sorted $U$ is assigned to $\tau$. We do this to pick a threshold $\tau$ such that $G_{\omega}$ can consider these samples it classifies correctly as “known” while leaving a small portion that are likely to be outliers out, when $G_{\omega}^{\mu}$ is compared with $\tau$. This approach could be further developed by also considering the true-positive and false-positive rates, however, we opted for a simpler approach in this study.

Finally, the reuse stage (lines 23-26) handles the execution of the imitated advice whenever appropriate, to aid the student in efficient exploration. It becomes active as soon as the imitation model $G_{\omega}$ is trained for the first time. Then, whenever $G_{\omega}^{\mu}(s_{t})<\tau$ (i.e. $G_{\omega}$ is familiar with $s_{t}$), no advice collection is occurred at $t$ and reusing is enabled for this particular episode, the student executes the imitated advice $\operatorname*{argmax}_{a}G_{\omega}(a\mid s_{t})$. Unlike [13], we do not limit advice reusing to the exploration stage of learning, e.g. the period $\epsilon$ is annealed to its final value in $\epsilon$-greedy. Instead, we define a reuse schedule that is independent than the underlying RL algorithm’s exploration strategy. At the beginning of each episode, the agent either enables reuse module with a probability of $\rho$ (set as $\rho_{init}$ initially). This value is decayed until it reaches its final value $\rho_{final}$ over $t_{\rho}$ steps, similarly to $\epsilon$-greedy annealing. This approach further eliminates the dependency of our algorithm to the RL algorithm’s exploration strategy.

We designed our experiments to answer the following questions about our proposal:

• •

  How does our automatic threshold tuning perform against the manually-set ones in terms reuse accuracy and learning performance?

• •

  How much does using advice imitation model to drive the advice collection process help with collecting more diverse state-advice dataset?

• •

  Does collecting a dataset with more diverse samples make any significant impact on the learning performance?

• •

  How much does every particular modification contribute in the final performance?

In the remainder of this section, we first describe our evaluation domain. Then, we provide the details our experimental process along with the substantial implementation details¹¹1The code for our experiments can be found at https://github.com/ercumentilhan/advice-imitation-reuse.

The results of our experiments in Enduro, Freeway, Pong, Q*bert and Seaquest are presented in several plots to let us analyse the performance of the student modes NA, EA, RA, AR, AR+A, AR+A+E, AIR in different aspects. Figure 2 contains the evaluation scores plots. In Figure 3, plots for the number of advice reuses per $100$ steps (top row) performed by AR, AR+A, AR+A+E, AIR; plots for the number of advice collections per $100$ steps performed by NA, RA, AR, AR+A, AR+A+E, AIR (middle row); plots for the values of the $\tau$ hyperparameter (uncertainty threshold) used by AR, AR+A, AR+A+E, AIR (bottom row) in a single run are shown. The shaded areas in these plots show the standard deviation across $3$ runs. The final evaluation scores, percentage of advice reuses in total number of environment steps as well as their accuracies are presented in Table II. Finally, in order to highlight the differences in the advice-collected state diversity of EA (identical collection strategy to AR, AR+A, AR+A+E), RA and AIR, these states from a single Pong run are visualised with the aid of UMAP [28] dimensionality reduction technique in Figure 4. Here, the scatter plot on the left compares AIR (blue) vs. EA (red) and the one on the right compares AIR (blue) vs. RA (pink); and the samples collected in common are shown in grey.

We first analyse the learning performances via evaluation scores. In Enduro, all methods but NA have a very similar learning speed and final scores, with AIR being slightly ahead of the rest. In Freeway, they all achieve nearly the same final scores, but they are distinguishable with small differences in learning speed where AR+A+E is on the top followed by other advanced student modes AIR, AR, AR+A. When we move to Pong, Q*bert and Seaquest, we finally see the student mode performances to be more distinctive. Even though the basic heuristics (RA and EA) show that a little number of advice from a competent policy can make substantial boosts on learning, these modes fall behind of those that employ advice reuse and fail to be a reliable choice, i.e. performing worse than NA in Freeway and Q*bert. Overall, the best mode AIR and the runner-up AR+A+E are ahead of all, with AR and AR+A following them.

Among the advice reuse approaches, we see that the most beneficial modification is the extended reuse schedule (+E) as it is highlighted by the difference between AR+A and AR+A+E. Defining such a schedule independently from the student’s RL exploration strategy involves using some extra hyperparameters, nevertheless, they are rather trivial to set arbitrarily. The trends in the advice reuse plots (Figure 3, top row) show how these schedules differ. The versions with +E (AIR and AR+A+E) yield around $10\times$ more reusing, which apparently plays an important role in the performance improvement. However, it is still not clear how to define the optimal reuse schedules.

Automatic threshold tuning (+A) also performs comparably, if not better, with the manual tuning approach (AR) as we can observe in the evaluation scores. Additionally, AR+A managed to achieve very similar reuse accuracies with AR; this also supports its success. When we examine the $\tau$ values, we see AR+A determined values that are close to the hand-tuned ones, except for the case in Pong where the difference is more significant. This reflected in reuse trend and evaluation performance, giving AR+A a very advantageous head start. These results support the idea that +A a far more preferable approach considering how problematic it can be to tune the sensitive $\tau$ threshold manually. For instance, if they were to deployed in some significantly different domains as they are, then we could potentially see AR+A coming far ahead of AR with a poorly tuned $\tau$. Furthermore, the periodic imitation model updates incorporated in AIR makes +A an essential component. In the bottom row of Figure 3, we also show how AIR changes its $\tau$ values over time as it collects more advice samples and updates its imitation model accordingly. Clearly, it is very difficult to manage these changes manually.

Finally, we also see that collecting advice by utilising the imitation model’s uncertainty (as it is done by AIR) contributes to the agent’s learning. When we look at the advice collection plots, we see $3$ different types of behaviours: early collection (EA, AR, AR+A, AR+A+E), random collection (RA), and AIR. Even though they seem to be occurring mostly in the same time windows, AIR does this in an uncertainty-aware fashion; hence the decreasing collection rate over time. Freeway is the case in which AIR is very selective. This is possibly due to the fact that in Freeway, the agent can traverse only a limited space which consequently reduces the diversity of the acquired observations. Nonetheless, this is not reflected in the evaluation scores as dramatically due to this game being rather trivial to solve. Another interesting observation is made by analysing the advice collected states in Seaquest by AIR, EA (which is identical to AR, AR+A, AR+A+E in terms of collection strategy), RA in a reduced dimensionality as seen in Figure 4. We chose Seaquest since it is the game where AIR is significantly ahead of AR+A+E, which can be credited to AIR’s only difference from it (collection strategy). We also include RA here mainly because it can potentially do better in acquiring different samples than EA. Here, the large circles denote the outliers (diverse samples) that are only covered only by either AIR, EA or RA. These are the important bits to pay attention to and compare. As it can be seen, AIR yields larger coverage, i.e. more diverse dataset of advice, in both cases against EA and RA collection strategies.

In this study, we proposed an automatic threshold tuning technique, an extended advice reusing schedule and an imitation model uncertainty-based advice collection procedure by extending the previously proposed advice reusing algorithm. We also developed a combined approach by incorporating these components, that is able to collect a diverse set of advice to build a more widely applicable advice imitation model for advice reuse.

The experiments in $5$ different Atari games from the ALE domain have shown that our enhancements provide significant improvements over the baseline advice reuse method as well as the basic action advising heuristics. First, being able to tune the uncertainty thresholds on-the-fly was observed to yield the learning performance of the carefully tuned threshold, which require unrealistic access to the tasks and extra effort to be adjusted. Secondly, we found that having the advice reusing process span across a larger portion of the learning session rather than just the steps that involve random exploration can yield superior performance. However, defining the best schedule for the maximum advice utilisation efficiency remains to be an open question. Thirdly, the uncertainty-driven advice collection method was found to be successful way to improve the imitation module’s dataset diversity. Nevertheless, periodic training process can be improved with better incremental learning techniques to make a better use of this simultaneous collection-imitation idea. Finally, our unified algorithm demonstrated state-of-the-art performance across $5$ Atari games by performing either on-par or better than its closest competitors.

The future extensions of this work can involve experimenting with more principled Policy Reuse approaches in the literature to further improve the advice reuse strategy. Furthermore, it will be a worthwhile study to make the teacher imitation better at learning online from the new samples it acquires. Finally, even though it is in the core motivation of our approach not to access and modify the agent’s RL components, it will be beneficial to investigate Learning from Demonstrations techniques and their possible contributions in our framework.

This research utilised Queen Mary’s Apocrita HPC facility, supported by QMUL Research-IT. http://doi.org/10.5281/zenodo.438045