Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learning

Most of agent-to-agent TL work is based on the teacher-student framework [22] where the teacher is an expert and the student is a novice agent. The student can request the teacher’s supervision, and due to resource constraints, their interaction is limited by a budget.

Preferred form of advice is action-based, i.e., [4, 5, 11, 13, 15, 17, 19, 21, 22, 28]. This lowers the overall exploration cost for the novice by asking for an action to follow in certain states. Other form of advice is Q-values [12, 13, 29] used to influence the action-selection process of a target agent. Finally, advice can be provided as policy to be followed for a certain number of steps [26] or a batch of RL experience, similar to a demonstration, as in [2, 3, 7, 14, 23].

Despite the good result shown by the teacher-student framework, roles are kept fixed and transfer impact is limited by the teacher knowledge. Therefore, insufficiently skilled expert might worsen novice’s performance. To overcome this limitation, [15, 26] introduce a real-time training of a centralised super-entity based exclusively on a subset of collected demonstration. This new entity provides on-demand advice when needed. While that improves advice quality over time, introducing a new super agent results in an additional cost to gather the experiences and to train the underneath model. [4, 11, 12, 19, 29] improved the base teacher-student framework by relying on confidence-based and importance-based methods to dynamically select one or more agents as teachers. In these models, an agent can ask for and provide advice simultaneously.

Baselines used in this paper are based on the above state-of-the-art methods: OCMAS is based on [11] and RCMP on [5]. In [11], an uncertain agent asks for advice and more confident agents recommend an action to follow. Confidence is measured through sars-RND and advice is given only when the advice seeker is the most uncertain agent across the team. Final action is taken by majority voting. In RCMP [5], an agent asks for advice during exploration based on its epistemic uncertainty. Advice, in form of an action, is given by a fixed expert-demonstrator. In our implementation, we replaced the single advisor with a jury of multiple trained agents to ward off any bias that might arise from using a single trained agent as teacher and action is selected by majority voting.

Table 1 summarises relevant recent work to this research. For each method, we indicate whether a prior external expert is needed or not. Note that we do not distinguish teacher expertise, i.e., optimal, sub-optimal, etc. When expert is not required, we specify whether the algorithm enables dynamic roles or not. Lastly, we report type of advice transferred and the frequency of transfer. The latter does not take into account when nor how often the advice is used.

Our work differs from the above as it removes the need for presence of an expert agent by dynamically selecting a learning agent as a teacher, to exploit different knowledge gained in different parts of a multi-agent system. Furthermore, while most of the related work continuously override target policy by advising actions, Q-values and policy, EF-OnTL combines local and received knowledge by transferring, less frequently, an experience batch specifically selected based on a gap between source and target beliefs. Lastly, EF-OnTL does not require the tuning of additional parameters, i.e., uncertainty threshold, to improve the quality of transfer.

In RL there are two types of uncertainties, aleatoric and epistemic. Aleatoric comes from the environment and is generated by stochasticity in observation, reward and actions. Epistemic uncertainty comes from the learning model and indicates whether the agent has adequately explored a certain state. Most recent TL frameworks rely on epistemic uncertainty to determine if an agent requires guidance.

A possible approach to approximating agent’s epistemic uncertainty in a particular task is to determine the frequency of visits to each state. In fact, [4, 15, 19] estimate uncertainty by relying on function defined over state visits counter. Similarly, [29] relies on number of visits over a state-action pair. However, state space might be continuous or very large making the counting unfeasible. Thus, state visit counts could be approximated by RND [1]. RND has been proposed to encourage exploration within agents but has already been exploited as uncertainty estimator for TL in RL in [2, 11]. RND consists of two networks, a target with an unoptimised and randomly initialised parameters and a main predictor network. Throughout time, the former is distilled within the latter and uncertainty is defined as prediction error between the two outputs.

Other sophisticated models can be used to estimate the epistemic uncertainty. [23] proposes uncertainty estimation through neural network, decision tree and Gaussian process. Although, neural networks seem to be preferred overall as are also used in [5], where the agent’s learning model is expanded by replacing control-layer with an ensemble to estimate agent’s uncertainty. However, despite the different underlying technique used to estimate uncertainty, all the presented methods rely uniquely on visited state.

To overcome this limitation, this paper introduces sars-RND as an extension of RND. sars-RND computes epistemic uncertainty over a full RL interaction $(s^{t},a^{t},r^{t},s^{t+1})$.

We propose State Action Reward Next-State Random Network Distillation (sars-RND)as an extension of RND to improve the estimation of epistemic uncertainty. When using RND as an uncertainty estimator, agent might lose important information that should be taken into account when computing uncertainty. For instance, in a sparse-reward environment and/or continuous control space, i.e, HFO and our Multi-Team Predator-Prey implementation described at Section 4, by not considering other details, such as action, agent might become confident within a state where the goal fulfilment is close but not yet achieved.

To overcome this potential limitation we propose sars-RND, an uncertainty estimator that takes into account a full RL interaction $(s^{t},a^{t},r^{t},s^{t+1})$, rather than just state, to compute epistemic uncertainty of an agent at a specific time.

This section outlines the core contribution of this work Expert-Free Online Transfer Learning (EF-OnTL). EF-OnTL is a novel online transfer learning algorithm that overcomes the need of an unique expert agent by dynamically selecting a temporary expert within each transfer iteration. Selected agent is used as source of transfer and some of its collected experience is made available to others. Target agent can then filter and sample a batch of experience to be integrated into its learning process and finally update its belief. Transferred batch contains five-elements tuples, $(s^{t},a^{t},r^{t},s^{t+1},u^{t})$, source agent’s RL interaction at time $t$ and $u_{t}$, an additional value that identifies source’s uncertainty over the RL interaction updated at time $t$ of visit.

EF-OnTL is independent of an underlying RL algorithm used Thus, it can be exploited on a range of RL methods, both tabular and neural network-based ones.

To define EF-OnTL framework, we specify the following:

• •

  $N$ $-$ number of RL-based agents available during a simulation;

• •

  a set of Agents $A=\{A_{1},\dots,A_{N}\}$;

• •

  a set of Learning Processes $LP=\{LP_{1},\dots,LP_{N}\}$ with the following relation $f_{1}={(A_{i},LP_{j})\in A\times LP\leftrightarrow i=j}$;

• •

  a set of Uncertainty Estimators $UE=\{UE_{1},\dots,UE_{N}\}$ and relation $f_{2}={(A_{i},UE_{j})\in A\times UE\leftrightarrow i=j}$;

• •

  a set of Transfer Buffers $TB=\{TB_{1},\dots,TB_{N}\}$ related to agents by $f_{3}={(A_{i},TB_{j})\in A\times TB\leftrightarrow i=j}$;

• •

  $B$ $-$ constrained Transfer Budget to be used within a single transfer interaction;

• •

  TF $-$ Transfer Frequency defined as number of episodes that occur between two consecutive transfer steps;

• •

  SS $-$ Source Selection technique used to select source of transfer;

• •

  TM $-$ Transfer Methods used to filter relevant knowledge to be transferred.

Algorithm 1 introduces high level procedure followed by agents for sharing experience one to another throughout their simultaneous exploration processes.

First, line $1$ defines EF-OnTL parameters. Then, at lines $3-6$ agents retrieve observation from an environment, sample an action based on their learning process and finally take a step. Afterward, each agent $A_{i}$ estimates its epistemic uncertainty $u_{i}^{t}$ over the interaction $(s_{i}^{t},a_{i}^{t},r_{i}^{t},s_{i}^{t+1})$, updates its $UE_{i}$ model based on new sampled evidence and publishes its uncertainty-labelled collected tuple to the associated transfer buffer $TB_{i}$ ($7-9$). When $TB_{i}$ is at full capacity, new labelled interactions replace the oldest tuples following a FIFO scheme. Learning process $LP_{i}$ is then updated based on the underlying model used, lines $10-12$. Lines $14-20$ show a transfer-step. At line $15$, source is selected among the agents w.r.t. some fixed criteria $SS$. Secondly, remaining agents, i.e., targets, apply a filtering $TM$ over source transfer buffer and then sample a batch composed by a fixed number $B$ of tuples ($17$). Finally, at line $18$, each target agent updates its learning process based on the obtained batch.

While the above provides a full description of the algorithm, Fig. 1 sketches EF-OnTL workflow for the simplest scenario with only 2 agents.

One of the most crucial parts of EF-OnTL is accurately selecting the source from which knowledge is extracted. In this section, we propose Source Selection Metrics (SS) to select a source agent among several candidates. Roles are dynamically assigned at the beginning of each transfer iteration and SS goal is to identify an experienced candidate whose knowledge will benefit others.

We propose two different methods to identify transfer source in  EF-OnTL: Average Uncertainty ($\overline{U}\lx@text@overline{U}$)and Best Performance (BP). Average Uncertainty ($\overline{U}\lx@text@overline{U}$)relies on average uncertainty $\overline{u_{i}}$ over transfer buffer $TB_{i}$. Therefore, source agent $A_{s}\leftarrow min_{i=0}^{N}(\overline{u_{i}})$ where, $\overline{u_{i}}~=~\frac{\sum_{t=0}^{|TB_{i}|}u_{i}^{t}\in TB_{i}}{|TB_{i}|}$. On the other hand, BP relies on performance achieved by agents during latest episodes. As performance measure we use average cumulated reward overs episodes’ finite-horizon undiscounted return ($\overline{R_{i}}$). $A_{s}\leftarrow max_{i=0}^{N}(\overline{R_{i}})$ where, $\overline{R_{i}}~=~\frac{\sum_{e=0}^{E}\sum_{t=0}^{\tau^{e}_{i}}r^{e,t}_{i}}{E}$. $E$ is the number of evaluated episodes and $\tau^{e}_{i}$ represents the length of the $e$-th episode for the $i$-th agent. Hence, average sum of rewards returned by the environment over episodes from initial to goal state with a finite number of steps.

Another crucial part of EF-OnTL is accurately selecting the knowledge to be transferred in each source and target transfer, as shown in line $17$ of Algorithm 1. In this section we introduce Transfer Filtering Criteria (TM) to prioritise certain tuples based on their score over a set of measures.

To simplify the readability of this section we describe the simplest case with two agents, which, at each transfer step, are referred to as source $A_{s}$ and target $A_{t}$. We rely on two criteria to identify experiences to be transferred, uncertainty and expected surprise. Expected surprise [25], is defined over target agent and is approximated through Temporal Difference-error (TD-error). For instance, given a DQN with predictor and target networks, TD-error is defined as Mean Squared Error (MSE) between the two outputs.

Assuming that $UE_{s}$ and $UE_{t}$ use a standard method with fixed parameters, we use their estimated uncertainties to select relevant tuples to be transferred. In detail, given ($o_{s}^{i},u_{s}^{i}$), an observation with its epistemic uncertainty explored by $A_{s}$ at time $i$, then $A_{t}$ estimates current uncertainty $u_{t}^{o_{s}^{i}}$ over the interaction $o_{s}^{i}$ sampled by $A_{s}$. Hence, $u_{t}^{o_{s}^{i}}=UE_{t}(o_{s}^{i})$. We then define $\Delta-$conf as discrepancy between the two estimations: $\Delta-$conf$=u_{t}^{o_{s}^{i}}-u_{s}^{i}$ . Note that $u_{t}^{o_{s}^{i}}$ is an epistemic estimation at the time of transfer and changes over time.

As a result, $A_{t}$ receives a personalised batch of experience that aims to fill shortcomings in its belief. Based on these two criteria we can define multiple filtering functions for incoming knowledge. In this paper, we introduced the following criteria to prioritise transferred experience:

• •

  rnd $\Delta$-conf $-$ Random transfer from Delta Confidence with threshold. By choosing this technique, an agent randomly samples $B$ interactions with an associated $\Delta-$conf above the median computed across all samples.

• •

  high $\Delta$-conf $-$ transfer Higher Delta Confidence. Agent selects the top $B$ entries with higher $\Delta$-conf.

• •

  loss $\&$ conf $-$ higher Loss and Confidence mixed. While previous filters are defined only over uncertainty, this one also considers expected surprise. To balance the different scales, values are normalised within a $[0,1]$ interval and then summed up. Finally, target agent pulls $B$ tuples from the buffer with the highest values.

The criteria presented above are explicitly or implicitly used in other TL frameworks. Nonetheless, we are the first to explicitly apply them to online transfer learning based on experience sharing, and evaluate EF-OnTL performance while exploring different combinations of SS and TM to assess their impact.

To assess sars-RND capability we use the MT-PP benchmark. Each estimator visits 2 states depicted by Fig. 2 while sampling different actions. We fixed architectures of both target and predictor across the two estimators, but their input layer dimensions differ as sars-RND requires extra neurons to accommodate action, reward and next state. Agents perform action $a_{0}$ for $250$ steps to strengthen the prediction of both uncertainty estimators. Afterward, action sampled is changed to $a_{1}$ for $25$ steps. Finally, agents resample $a_{0}$ for $25$ steps.

Overall, we observed higher uncertainty within the first steps while curves decrease asymptotically to $0$ for both estimators as they increasingly visit the states. However, by evaluating uncertainty in a narrowed interval as in Figure 4, we notice that while RND keeps a flat trend, sars-RND registers a spike in uncertainty on the change of action, firstly, at step $250$ when action $a_{0}$ is replaced by $a_{1}$, and secondly, at $275$ when $a_{0}$ is resampled. Second spike is smaller compared to first as sars-RND recognises the previous seen interaction.

This confirms thats sars-RND provides a more reliable uncertainty estimation by considering executed action alongside visited state while RND is not sensitive to action variation. Furthermore, given the observed decreasing trend, sars-RND generalises well enough to recognise familiar states even while sampling a different action. Thus, uncertainty estimated for previously un-encountered state is higher than previously seen state while exploring different actions.