Herd’s Eye View: Improving Game AI Agent Learning with Collaborative Perception

In autonomous vehicle research, the bird’s-eye view semantic segmentation task involves predicting pixel-level semantic labels for a top-down ego-centric view of an environment. Segmentation classes are typically dedicated to vehicles, driveable areas, and obstacles. In prior BEV research, a significant point of distinction lies in the method used for transforming 2D perspective-view features into 3D space or directly onto the BEV plane. Many previous works have leveraged explicit geometric reasoning in their perspective transformation (Reiher, Lampe, and Eckstein 2020; Philion and Fidler 2020; Hu et al. 2021). An approach that has recently gained popularity is the Cross-View Transformer (CVT) (Zhou and Krähenbühl 2022) model, which implicitly models scene geometry. The CVT model leverages a camera-aware cross-view attention mechanism to implicitly learn a mapping from individual camera views to a canonical map-view representation for map-view semantic segmentation. The model consists of a convolutional image encoder for each view and cross-view transformer layers for inferring the segmentation in a simple, easily parallelizable, and real-time manner. BEVFormer (Li et al. 2022c) uses a similar cross-attention model to extract spatiotemporal BEV information. BEVSegFormer (Peng et al. 2022) uses a deformable transformer-based encoder. There are many publications in this research area using similar architectures of transformers to shift perspective view(s) to BEV, Ma et al. provides a recent review of these architectures.

The HEV semantic segmentation task poses a unique challenge compared to the BEV task since the agent translations are unknown; this requires the model to geometrically reason about multiple camera views to localize. For our baseline approach, we leverage the CVT model proposed by (Zhou and Krähenbühl 2022). The CVT model is well suited for the HEV task because of its global attention mechanism. Many BEV publications such as BEVFormer (Li et al. 2022c) and BEVSegFormer (Peng et al. 2022) aim to optimize this global attention mechanism since in ego-centric tasks, a camera view only overlaps with a consistent subsection of the map-view. Conversely, in our HEV world-centric use case, global attention is an advantage because a camera view can overlap with any part of the map-view. Additionally, we expect that the model’s performance can be further improved by incorporating additional information from other sensors, such as lidar and radar, as demonstrated by recent works (Harley et al. 2022).

Autonomous vehicle datasets have been widely used in collaborative perception research, comprising various sensory inputs, including cameras, lidar, and GPS (Han et al. 2023), from multiple vehicles in a vehicle-to-vehicle environment (Xu et al. 2022c; Yu et al. 2022). Some datasets, such as those proposed in (Li et al. 2022a; Mao et al. 2022), include infrastructure sensors, resulting in a vehicle-to-infrastructure data model. Others, such as the dataset presented in (Xu et al. 2022b), employ a vehicle-to-everything model. The CoPerception-UAVs dataset (Hu et al. 2022) employs five images from five drones flying in formation. It is worth noting that these datasets are all sourced from CARLA (Dosovitskiy et al. 2017) in Unreal Engine, a widely used open-source platform for simulating autonomous vehicles.

The HEV datasets sourced from our simulated environments are uniquely challenging in the field of collaborative perception, as the agents are equipped with only one or two cameras. Unlike previously proposed collaborative perception problems, the HEV task does not provide the agents with the transformation component of their pose. The unknown position of each camera view within the global coordinate frame adds a significant challenge to the semantic segmentation prediction task and other downstream tasks.

Collaborative perception has been explored in recent years, improving the capability of single-agent perception models (Li et al. 2022b; Hu et al. 2022; Lei et al. 2022; Su et al. 2023; Zhou et al. 2022). In conventional collaborative perception, intermediate representations produced by sensors or neural networks from multiple viewpoints are propagated among a team of robots, such as a group of vehicles (Hu et al. 2022; Xu et al. 2022a) or a swarm of drones (Zhou et al. 2022; Hu et al. 2022). The existing works commonly learn a collaboration module, produced by a Graph Neural Network (Zhou et al. 2022; Li et al. 2021), Convolutional Neural Network (Li et al. 2022b; Qiao and Zulkernine 2023), or a Transformer (Xu et al. 2022a; Hu et al. 2022) to combine multiple robot intermediate representations.

Prior research has focused on robots equipped with multiple sensors, requiring sensor data fusion on a per-agent basis before information exchange among agents (Han et al. 2023). However, in this work, we focus on robots with only one or two cameras and no additional sensors, making our approach more amenable to smaller, simpler robot swarms. Since we focus on simpler robots, we do not utilize a collaboration module, and instead fuse all camera views together in a single cross-attention module.

In the Herd’s Eye View (HEV) semantic segmentation task, we are given a set of $n$ monocular camera views, $(I_{k},K_{k},R_{k})^{n}_{k=1}$ consisting of an input image $I_{k}\in\mathbb{R}^{H\times W\times 3}$, camera intrinsics $K_{k}\in\mathbb{R}^{3\times 3}$, and extrinsic rotation $R_{k}\in\mathbb{R}^{3\times 3}$ with respect to the agent base. The goal of the HEV task is to predict a binary semantic segmentation mask $y\in\{0,1\}^{h\times w\times C}$ in the global coordinate frame, where $C$ is the desired number of segmentation classes. The HEV task adds additional ambiguity to the well-studied BEV task as each camera view is at an unknown translation and orientation with respect to the global coordinate frame.

We define a BEV as a single-agent perception transformed into an ego-centric view, whereas the HEV is a collaborative perception transformed into a fixed world-centric view. A comparison of the ego-centric views tested and the fixed word-centric view can be seen in Figure 3.

Our approach, seen in Figure 1, follows three steps:

1. 1.

   Collect multiple views of the environment from robot cameras.

2. 2.

   Use a collaborative perception model to obtain the HEV, the world-centric semantic segmentation of the environment.

3. 3.

   Input the HEV to a Reinforcement Learning (RL) control policy to obtain agent control commands.

Our goal is to establish a baseline HEV perception model to extract information from the multiple camera views and project them onto a fixed world-centric view. We propose a baseline perception model using the Cross-View Transformer (CVT) (Zhou and Krähenbühl 2022) and use semantic segmentation as our downstream task. The Cross-View Transformer is a recent approach that uses a cross-view attention module, first proposed by (Zhou and Krähenbühl 2022), enabling the agents to reason about their environment in an ego-centric coordinate frame. We extend the CVT model to further improve its accuracy and speed for the HEV use case. We name our baseline model the Herd’s Eye View Cross-View Transformer (HEV-CVT). We use a world-centric map embedding and tune positional embeddings, output sizes, and the number of transformer layers to fit our proposed HEV environments.

We use identical Unity simulation environments to source the datasets for training the HEV semantic segmentation task and MARL task. To collect the HEV ground truth for both tasks, we use our own custom fork of MBaske’s Unity Grid-Sensor Library (Baske 2021) which allows the collection of HEV world-centric grid-sensors. The only difference between ego-centric based agents and world-centric based agents is the location of their grid-sensor and the perspective at which they take their actions (e.g., forward for the word-centric agent is always North, but forward for the ego-centric agent is with respect to their current orientation). All agents are trained on ground-truth sensors, calculated using the bounding boxes that are individually tuned to each object. The resolution of the grid-sensor is adjusted to accommodate the complexity and size of the environment as seen in Table 1. Example observations of world-centric and ego-centric based agents can be seen in Figure 3

Our simulations are conducted in three different Unity ML-Agents environments:

Collaborative Push Block: Three agents are required to push white blocks to a green goal area on a randomly selected side of a square area. There are blocks sized one, two and three, each requiring the respective amount of agents to push into the goal area (Cohen et al. 2022).

Dungeon Escape: As a Green Dragon slowly walks towards an exit portal, one of the three agents must collide with it in order to sacrifice itself and spawn a key. The key must then be picked up by one of the two remaining agents and brought to the exit door to escape the dungeon (Cohen et al. 2022). Once any agent escapes, all agents win.

Planar Construction: Six agents collaborate to push red pucks into desired positions. Desired positions are randomly assigned to valid coordinates within the arena, and are observed via a Grid-Sensor, similar to the Push Block environment (Strickland, Churchill, and Vardy 2019). In each round a new random amount of pucks from 2 to 16 are spawned.

We utilize the open-source Collaborative Push Block and Dungeon Escape environments from ML-Agents (Cohen et al. 2022), which are already native to Unity and only change the sensor input of agents. We recreate the Planar construction task (Vardy 2018; Strickland, Churchill, and Vardy 2019; Vardy 2020, 2022, 2023) based on Strickland, Churchill, and Vardy’s work in the CWaggle simulator but adapt the environment to Unity ML-Agents. All three environments can be seen in Figure 2. For MARL training, we use the HEV ground truth as model input and identical reward functions to the original implementations. Specifically, the agents are trained using the Multi-Agent POsthumous Credit Assignment (MA-POCA) algorithm (Cohen et al. 2022) in Unity ML-Agents. By using identical reward functions, we aim to create a fair comparison between the performance of agents using HEV and those using traditional sensor frames in cooperative scenarios.

The MARL task enables us to train the CVT models, which can perform semantic segmentation in an ego-centric or world-centric view. To collect the data necessary for training the CVT models, we run the trained MA-POCA models and collect the camera view, camera intrinsics, and rotation extrinsic from each agent at each step of the simulation, along with the ground truth HEV and BEV. By collecting data from various environments and introducing variations, we aim to create diverse and robust datasets for training the CVT models.

The Cross-View Transformer is adapted from Zhou and Krähenbühl for the Herds Eye View Collaborative Perception task. The first stage of the network passes each input image from agents into a feature extractor, we use an EfficientNet-B4 (Tan and Le 2019), which outputs two multi-resolution patch embeddings of size (28, 60) and (14, 30). Each patch is passed into a Cross-View Transformer convolution stack as in the original implementation. We found fewer convolution stacks significantly degrade the HEV-CVTs ability to localize, and more are not necessary. The patch embedding act as image features and are used in the keys and as the values for the Cross-View Transformer.

We encode the rotation $R_{k}\in\mathbb{R}^{3\times 3}$ of the agent’s camera into a $D$-Dimensional positional embedding using a multilayer perceptron. We use $D=64$ for all of our experiments. The positional embedding is combined with the image feature to compute the keys for the cross-view transformer. The world-centric map embedding operates similarly to the originally proposed map-view embedding. The key difference with our approach is we do not subtract camera location embeddings from the map embedding, instead, we directly use the learned map embedding as queries. The camera locations with respect to the world are unknown for the HEV task, and we found subtracting rotation embeddings did not improve performance. The transformer architecture refines its world-centric estimate through two rounds of computation, each resulting in new latent embeddings used as queries.

The cross-view transformer computes softmax-cross-attention (Vaswani et al. 2017) between the image feature keys, values and world-centric queries. This setup allows world coordinates from the world-centric map embedding to attend to one or more image locations, allowing the model to reason about the environment from multiple image features. The multi-head attention mechanism uses 4-heads like the original implementation but with half the embedding size of $d_{head}=32$.

The cross-view transformer output is 8x8 for square environments and 8x8 and 8x16 for rectangular environments, this then passes through a decoder consisting of three up-convolutional layers to a final size of 64x64 and 64x128. This is purposely larger than is required for RL observation size, as smaller sizes can create ambiguity for some object occupancy resulting in decreased performance. These larger HEV-CVT sizes can easily be down-sampled to match the required RL observation sizes of 32x32 and 32x64. We threshold the output prediction confidences, keeping predictions with a confidence greater than 0.4. The prediction confidences prior to thresholding can be seen in Figure 4 as a heat map (lighter is higher confidence).

Our training process is similar to the original implementation by Zhou and Krähenbühl, we also use focal loss (Lin et al. 2017) and the AdamW (Loshchilov and Hutter 2017) optimizer with a one-cycle learning rate scheduler (Smith and Topin 2018). All models are trained with a batch size of 4 for 25 epochs. Training lasts approximately 8 hours on a single RTX 3090 GPU before converging.

The HEV-CVT model must accurately localize the position of each agent based on the overlap of camera frames, which are located at unknown positions. An example of this in the Dungeon Escape environment can be seen in Figure 4. The cameras are recorded at resolution $480\times 224$, and we use the camera intrinsics of a Raspberry Pi Camera Module 3. Consistent with prior works (Ma et al. 2022), we show the result Intersection over Union (IoU) metric for the HEV-CVT model trained on each environment in Table 2. We compare the performance of the baseline CVT model on world-centric, ego-centric, and ego-forward coordinate frames.

In the Collaborative Push Block environment, three agents are equipped with two forward-facing cameras and are tasked with predicting the occupancy of all push blocks, agents and the goal area. In the Dungeon Escape environment, three agents are equipped with two forward-facing cameras and are tasked with predicting the occupancy of the dragon, agents and key. In the Planar Construction environment, six agents are equipped with a single forward-facing camera and are tasked with predicting the occupancy of all pucks and agents.

Our results shown in Table 2 demonstrate the world-centric coordinate frame consistently outperforms the ego-centric coordinate frames in all environments. The Collaborative Push Block and Dungeon Escape environments show the largest performance improvements, with up to 32.72% and 17.46% improvement in IoU, respectively. These results suggest that the world-centric HEV approach is effective in addressing the challenges of collaborative perception in multi-agent environments. This result is especially apparent in the Collaborative Push Block environment, where the HEV-CVT model easily localizes itself based on the large goal location seen in most camera views for a near-perfect 96.94% IoU score. The landmarks in the Dungeon Escape environment, the exit door and portal are in randomized locations which makes localization harder than the Push Block environment, reflected by the steep drop in IoU scores.

The standard ML-Agents environments were not as challenging for the CVT models as there were not many permutations of the environment layout. By contrast, our custom Planar Construction environment presents a more complex challenge as we randomly change the coloring of six wall and floor components at every time step of the environment during data collection. Additionally, the locations of pucks to be pushed are randomized, and the environment area is twice the size of the ML-Agents environments. Despite the additional challenge the HEV-CVT model still performs well in the Planar Construction environment scoring 48.37% on the HEV task. This result shows the CVT models can localize based on the overlap in views between cameras as much of the validation set contains wall colors and puck layouts never before seen.

In order to compare the performance of the fixed world-centric coordinate frames with other commonly used coordinate frames, we conduct experiments in all three proposed environments. To ensure a fair comparison between the performance of agents using different coordinate frames, we use identical reward functions to each environment’s original implementation and identical grid sizes.

Table 3 compares the performance of agents using different coordinate frames in all three proposed environments. We find consistently lower episode lengths with world-centric based agents compared to ego-centric. We opt to use episode length as our performance metric, as it directly reflects the speed of task completion. While alternative metrics such as cumulative or mean reward are also commonly used, these primarily reflect minor negative rewards assigned per time step, providing less insight into an agent’s efficiency in our context.

Our experiments highlight a common challenge faced by BEV-based agents in all three environments, often an object necessary to take the optimal action was missing from the agent’s view, leading to sub-optimal decision-making and increased episode lengths. This was especially apparent in the Push Block environment where often one of the three agents would not observe the size three block (requiring all three agents to push it), causing two agents to be waiting for the third agent to join them at the block, wasting time. Conversely, we found HEV-based agents in the Push Block environment stuck close together and consistently pushed the highest value blocks together first.

The HEV-based agents were able to leverage the multiple viewpoints available to them, enabling them to better perceive their environment and take more optimal actions. This issue was particularly evident in the Push Block environment, where the improved perception of world-centric agents resulted in significantly lower episode lengths than ego-based agents.

Overall, these findings suggest that the HEV framework offers a superior perception model in MARL environments, providing agents with a more comprehensive understanding of their surroundings, leading to improved decision-making and better overall performance.