Multi-Task Multi-Agent Shared Layers are Universal Cognition of Multi-Agent Coordination

A pre-trained model is a model that’s already well-trained, and it shares its knowledge with a new model to speed up the new model’s training, without the need to start from scratch. Pretrained models [9, 10] extract common knowledge of a broad range of tasks from data. It can accelerate and improve training of downstream tasks, by leveraging the common knowledge stored in them in various means. Pre-trained models can be applied to a wide range of downstream tasks, such as image classification [11, 12, 13, 14], object detection [15, 16, 17, 18, 19], text classification [20, 21, 22, 23, 24] and more. When training for downstream tasks, we need to load the parameters of the pre-trained model’s feature layers. After that, keeping the feature layers (the initial layers) fixed and fine-tuning the parameters of the later layers [25] because the initial layers primarily focus on extracting generic information. For instance, in ResNet, the early layers are responsible for capturing low-level features like textures and shapes.

Traditionally, pre-trained models are usually trained through supervised learning (SL) on large data sets. They possess extensive language and visual knowledge, making them directly applicable to new tasks or amenable to fine-tuning for specific tasks. Using pre-trained models significantly reduces the time and computational resources required to build and train models from scratch, especially when data for new tasks is limited. In supervised learning, especially in the field of image processing, we often use large labeled datasets like ImageNet [26], Open Images dataset [27], and COCO [28] to pre-train models and obtain large pre-trained models such as AlexNet [29], VGG [30], and ResNet [31]. These pre-trained models typically learn generic features that can be useful for many downstream tasks. These features speed up the process of learning specific characteristics for the downstream tasks, making the overall training faster.

Because the cost of data annotation is too high, researchers use self-supervised learning (SSL) methods to train pre-trained models. Up to now, self-supervised pre-trained models are being applied in various domains, including NLP [20, 32, 33, 34], computer vision [35, 36, 37, 38], and audio [39]. Unlike supervised learning, self-supervised pre-training doesn’t require labeled data. There are two main approaches: one involves creating self-generated tasks to learn feature representations from the data, such as tasks like image completion, text generation, image rotation, and image augmentation. The other approach uses contrastive learning to bring similar samples closer in feature space, thus capturing common underlying features among similar samples.

Inspired by large-scale language modeling, DeepMind has applied a similar approach to create a unified intelligent agent called Gato [40]. Gato has been trained on over $600$ different tasks, each with varying patterns, observations, and action guidelines. However, Gato, despite integrating a variety of decision tasks, still lacks generalization abilities. This is because in the single-agent domain, different decision tasks require completely different knowledge. For example, in Atari games, tasks like pong and breakout have no shared knowledge between them. In contrast, in multi-agent tasks, such as $3s5z$ to $3s5zvs3s6z$ in SMAC, there exists strong shared knowledge, such as teammate attributes, enemy attributes, strategies like focus fire, hit and run, and more. In multi-agent reinforcement learning, there typically exists significant variation in the state space among different tasks. Current pre-trained methods yield universal perceptual models that cannot be shared across diverse multi-agent tasks. Different multi-agent tasks often involve more shared decision knowledge, such as agent cooperation. Therefore, we propose a multi-task, multi-agent pre-training approach aimed at acquiring pre-training models that encompass shared decision knowledge.

We refer to the front-end network layer as the Perception Layer (PerL) and the back-end network layer as the Decision Layer (DecL). During the pre-training phase, we simultaneously train $N$ multi-agent tasks. For each $task_{n}$, $n=\{1,2,\cdots,N\}$, we collect data generated by the interaction between agents and the environment in $task_{n}$ and store it in the replay buffer $Buffer_{n}$. We then sample from $Buffer_{n}$ for training. Samples are input into ${\rm PerL}_{n}$ to extract perceptual features, followed by using DecL to predict $action_{n}$. Each $task_{n}$ has its dedicated ${\rm PerL}_{n}$ while sharing DecL. Once pre-training is complete, we obtain DecL containing common decision knowledge among different tasks. We fix the parameters of the pre-trained DecL and transfer it to a new task, subsequently training PerL for that task. This approach efficiently allows us to learn effective policies for new tasks, leading to a significant reduction in training costs.

Value-based MARL algorithms, such as VDN and QMIX, estimate state-action values by taking the state $s$ as the network input and generating estimated values for each action $a_{i}$ under that state. The network’s output size remains fixed at the size of the task’s action space. Due to variations in action spaces across different multi-agent tasks, sharing the back-end network layer between tasks is infeasible. To address this challenge, we position the action $a_{i}$ at the forefront of the network’s input. The network takes state-action pairs as input and yields corresponding value estimates as output. This structural design maintains a fixed network output size of $1$. By utilizing front-end network layers to map inputs from different tasks to a consistent dimension $d$, we enable sharing the back-end network layer across different tasks. Pairing state and all actions as inputs to the network results in value estimates for all state-action pairs. This network architecture is referred to as the Action Prepositioning Network (APN).

Based on APN, we can share DecL across different tasks during multi-task pre-training. We will illustrate the training process of any task $task_{n}$ using APN at time step $t$. For simplicity, we omit the $t$ subscripts for all variables. $task_{n}$ comprises $M$ agents, each with an action space size of $K$. $o_{m}$ represents the observation of agent $m$, $a_{m}^{{}^{\prime}}$ is the action executed by agent $m$ in the previous time step, $id_{m}$ is the identifier of agent $m$, and $a_{m,k}$ represents the $k$-th executable action of agent $m$, $m=\{1,2,\cdots,M\}$, $k=\{1,2,\cdots,K\}$. For each agent $m$, perceptual features $f_{m}$ are extracted from the task’s input using independent ${\rm PerL}_{n}$:

where $f_{m}\in\mathbb{R}^{K\times d}$, with $d$ representing the dimension of perceptual features. Given that each agent can only partially observe the environment at each time step, we employ the GRU recurrent neural network in PerL to encode historical trajectories and address the issue of partial observability. These $f_{m}$ are inputted into the shared DecL to predict value estimates $Q_{m}$:

where $Q_{n}\in\mathbb{R}^{K\times 1}$. We select the maximum Q value, denoted as $Q_{*}$, for each individual agent and aggregate them through a mixer to predict the total Q value $Q_{\rm predict}$. In this context, we employ a mixer the same as VDN. We input the same data into the target APN network to generate the target total Q value $Q_{\rm target}$. The APN is updated by minimizing the error between $Q_{\rm predict}$ and $Q_{\rm target}$.

During multi-task training, not all tasks converge at the same pace. Typically, simpler tasks tend to converge faster than complex ones. To enhance the efficiency of multi-task training, it becomes essential to allocate larger loss weights to those tasks that exhibit slower convergence. We standardize evaluation metrics across different tasks. $EM_{n}$ represents the normalized evaluation metric value for $task_{n}$ and is used to assess the current convergence status of $task_{n}$. $EM_{n}^{-}$ represents the inverse evaluation metric of $EM_{n}$, such as the negative or reciprocal value of $EM_{n}$. A smaller $EM_{n}$, or equivalently, a larger $EM_{n}^{-}$, indicates poorer convergence for the corresponding task. We introduce a method for dynamically and adaptively assigning weights to multi-task losses, wherein the weights are adjusted as $EM_{n}^{-}$ change during the training process. Here, $loss_{n}$ corresponds to the loss of $task_{n}$, $\omega_{n}$ represents the weight of $loss_{n}$, and the total loss $total\_loss$ for all tasks is calculated as follows:

We increase the number of tasks used for pre-training to explore whether incorporating more tasks can lead to improved acquisition of common knowledge. All our experiments were conducted specifically on the hard and super-hard difficulty levels of the SMAC task. The tasks involved in the experiments are detailed in table 1.

Three-task pre-training. We employ two sets of tasks for pre-training DecL: $Group_{1}:(task_{1},task_{3},task_{5})$ and $Group_{2}:(task_{7},\\ task_{8},task_{9})$. All tasks within $Group_{1}$ are of hard difficulty, while those in $Group_{2}$ are of super hard difficulty. The DecL pre-trained with $Group_{1}$ is transferred and separately fine-tuned for PerL training on $task_{7}$ and $task_{10}$. Similarly, the DecL pre-trained with $Group_{2}$ is transferred and separately fine-tuned for PerL training on $task_{3}$ and $task_{4}$. Experimental results are depicted in Fig. 2.

From Fig. 2 (a) and (b), it is evident that the APN network, during multi-task training, effectively learns strategies for each task. The DecL obtained from multi-task pre-training encompasses decision knowledge utilized in all tasks, namely, generic decision knowledge. Fig. 2 (c) shows that using the DecL pre-trained on three hard tasks results in faster convergence for $task_{7}$ compared to training from scratch. Fig. 2 (d) demonstrates that, with the DecL pre-trained on three hard tasks, $task_{10}$ achieves superior convergence compared to starting from scratch. Fig. 2 (e) shows that utilizing the DecL pre-trained on three super hard tasks leads to faster convergence and better final convergence results for $task_{3}$ compared to starting from scratch. However, Fig. 2 (f) reveals that, despite being pre-trained on three super hard tasks, $task_{4}$ exhibits convergence results and rates consistent with training from scratch, indicating no improvement. Clearly, in comparison to DecL pre-trained on a single task, DecL pre-trained on three tasks shows better performance when transferring to new tasks. This highlights the advantage of having more pre-training tasks in obtaining DecL with more comprehensive generic decision knowledge. The lack of improvement in the convergence performance of $task_{4}$ also suggests that the decision knowledge contained within DecL can only cover some tasks comprehensively, prompting further consideration of increasing the number of pre-training tasks.

Six-task pre-training. We select six tasks for pre-training DecL:
$Group_{3}:(task_{1},task_{2},task_{5},task_{6},task_{7},task_{8})$. We fix the pre-trained DecL parameters and transfer them separately to $task_{4}$ and $task_{10}$. The experimental results are depicted in Fig. 3.

Fig. 3 (a) demonstrates that our proposed method continues to acquire effective policies across all tasks when the number of pre-training tasks increases to six. In Fig. 3 (b), we observe that training $task_{4}$ with DecL pre-trained based on $Group_{3}$ tasks results in faster convergence and superior final performance compared to training from scratch. Furthermore, employing DecL pre-trained on $Group_{3}$ tasks for $task_{4}$ converges more rapidly than using DecL pre-trained on $Group_{2}$ tasks. The same result can also be found in Fig. 3 (c). Training $task_{10}$ with DecL pre-trained on $Group_{3}$ leads to faster convergence and improved outcomes compared to using $Group_{1}$ pre-trained DecL. In summary, through multi-task pre-training, we can extract DecL containing common decision knowledge, which becomes more generalized with increased pre-training tasks.

VDN to QMIX. To further validate the inclusion of common decision knowledge in the DecL pre-trained with the VDN algorithm, we migrate the pre-trained DecL from three hard difficulty and three super hard difficulty SAMC tasks to $task_{3}$ and $task_{7}$, respectively. We employ the QMIX algorithm to train PerL on these new tasks. Experimental results are presented in Fig. 4.

Fig. 4 demonstrates that when the DecL pre-trained on three tasks using the VDN algorithm is transferred to new tasks and trained with the QMIX algorithm, the final convergence results are superior to those of the QMIX algorithm trained from scratch. Consequently, the DecL obtained from multi-task pre-training indeed encompasses common decision knowledge. This common decision knowledge is applicable to various new tasks as well as new multi-agent reinforcement learning algorithms.

APN-VDN vs VDN. To enable multi-task training, we initially place the action at the front of the network input and maintain this structure when transitioning to new tasks. To illustrate the impact of adopting this structure on experimental performance, we train original VDN and APN-VDN on all SMAC tasks, with the results presented in Fig. 5 and table 2.

As shown in Fig. 5, adopting the APN structure has a slight impact on the convergence speed of the original VDN but does not affect the ultimate convergence results. Table 2 demonstrates that APN-VDN and original VDN achieve nearly identical convergence results across all SMAC tasks. Therefore, employing the APN structure does not influence the performance of multi-agent algorithms. Consequently, we use APN-VDN as the baseline algorithm for comparison in our experiments, with variations in algorithm performance stemming from the utilization of pre-trained DecL.

Single-task Pre-training. We conduct experiments on $task_{1}$, $task_{2}$, and $task_{9}$. We perform pre-training on $task_{2}$ and $task_{9}$ to obtain DecL with task-specific decision knowledge. Subsequently, we freeze the parameters of DecL and transfer them to $task_{1}$ for retraining PerL. Experimental results are illustrated in Fig. 6.

Fig. 6 reveals that when using DecL pre-trained on $task_{2}$ and $task_{9}$, the final convergence outcomes for $task_{1}$ are consistently worse compared to training from scratch. This occurs because pre-training DecL on a single task leads to overfitting on that specific task, preventing the acquisition of general decision knowledge. Consequently, when we transfer a DecL pre-trained on a single task to a new task, it fails to align with the new task, resulting in a deterioration of the final convergence performance for the new task. Therefore, we need to use more pre-training tasks to extract DecL containing general decision-making knowledge.

Fix vs Fine-tune. When employing pre-trained models, we typically face the choice between weight fixation and fine-tuning. Our experiments explore both scenarios when transferring pre-trained DecL weights on $Group_{1}$ tasks to $task_{8}$. The experimental results, as depicted in Fig. 7, demonstrate that fixing the pre-trained DecL weights leads to efficient convergence to superior results. During the backpropagation process, the loss diminishes as it propagates from the network’s output to the input. DecL is closer to the network’s output, resulting in more significant parameter updates compared to PerL. In the early stages of training, PerL has not yet developed robust perceptual capabilities. Consequently, the transmission of substantial losses can disrupt the pre-trained DecL, leading to decreased convergence speed and inferior results. Thus, we consistently opt for the fixed-weight approach when employing pre-trained DecL.

To better illustrate how our proposed multi-task multi-agent reinforcement learning algorithm learns to share common decision knowledge between different tasks, we visualize the acquired policies in the SMAC tasks. The videos of the battle are provided in the supplementary material.

In Fig. 8(a) and (d), we visualize the performance on $task_{2}$ after training for $6$ million steps with and without the pre-trained DecL using $Group_{2}$ tasks. In Fig. 8(a), the attack positions of all Marines remain nearly unchanged, displaying a lack of flexibility, even when their health is low. However, in Fig. 8(d), the Marines exhibit a more flexible policy by moving to disperse enemy fire and concentrating their attacks on a portion of the enemy forces.

Fig. 8(b) and (e) visualize the performance on $task_{7}$ after training for $10$ million steps without pre-trained DecL and with DecL pre-trained on $Group_{1}$ tasks, respectively. In Fig. 8(b), the Medivac does not employ an evasive strategy, making it vulnerable to enemy attacks, resulting in losses to our Medivac, while the health of other agents cannot recover promptly. Additionally, the positioning of Marines and Marauders lacks strategic depth, failing to execute timely retreats when their health is low. However, in Fig. 8(e), the agents have acquired advanced policies: Marines with low health retreat promptly, while healthier Marines draw enemy Marauders’ fire, allowing other Marines to flank the enemy Marauders for surprise attacks stealthily.

In Fig. 8(c) and (f), we visualize the performance on $task_{4}$ after training for $0.6$ million steps, comparing scenarios with and without pre-trained DecL from $Group_{3}$ tasks. In Fig. 8(c), all Banelings charge towards the opponent to detonate them, while some Zerglings also advance but are damaged by the Banelings’ explosions. However, in Fig. 8(f), all Banelings immediately rush towards the opposing camp, while Zerglings quickly retreat to avoid unintentional harm from the Banelings, resulting in minimal casualties.

In summary, by utilizing a pre-trained DecL incorporating generic decision knowledge to train for new tasks, we typically acquire more effective policies within the same training steps, thus achieving superior results more efficiently.

We conduct experiments on six tasks from GRF, as shown in table 3. We choose four tasks, including both easy and hard difficulties $(task_{11},task_{13},task_{14},task_{15})$, for pre-training the DecL. Subsequently, we transfer the pre-trained DecL to $(task_{12}$ of easy difficulty and $(task_{16}$ of super hard difficulty. The experimental outcomes are depicted in Fig. 9.

Fig. 9 (a) illustrates that the APN network, during multi-task pre-training, effectively learns policies across various GRF tasks, each involving a different number of agents. Fig. 9 (b) demonstrates that with the pre-trained DecL, the convergence speed and final convergence results for $task_{12}$ of easy difficulty significantly improve compared to training from scratch. Fig. 9 (c) reveals that, with the pre-trained DecL, the final convergence results for $task_{16}$ of super hard difficulty are superior to those obtained from training from scratch. The utilization of pre-trained DecL renders training on new GRF tasks more efficient. Notably, $task_{16}$ fails to acquire effective policy when trained from scratch. In summary, through multi-task pre-training, we can obtain a DecL containing common decision knowledge, which enables us to more efficiently acquire policies for new tasks, substantially reducing training costs.