GNN-based Decentralized Perception in Multirobot Systems for Predicting Worker Actions

In manufacturing setups, numerous dynamic processes such as machining, assembly, and material handling occur simultaneously. Human workers perform multiple tasks like machine tending and assembly, requiring adaptability. These dynamic conditions pose significant challenges for robots, which must operate efficiently while prioritizing human safety.

Although manufacturing represents just one subset of potential robot deployment scenarios, it serves as a baseline and motivation for designing our multi-robot system. We define intent prediction as the anticipation of a human operator’s future action within a specific time horizon. For this study, we consider time horizons of 1s, 2s, and 3s into the future, based on the Stopping time and distance metric outlined in ISO 10218-1:2011 safety standards, Annex B(normative) [27]. This metric relates to the time required for a robot to detect a human, initiate deceleration, and come to a complete stop, which establishes the minimum threshold for the perception pipeline’s prediction capabilities. Given the relatively low speeds of industrial mobile robots¹¹1MiR250 [28] and Otto 1500 [29] both have a top speed of 2.0 m/s, the selected time horizons are deemed sufficient for safety and operational efficiency. For instance, if a human is detected at a distance of 8 meters while both the robot and the human are moving towards each other, the robot has less than 2 seconds to react²²2Assuming a combined closing speed of 4.1 m/s (robot at 2.5 m/s and human at 1.6 m/s), the time to collision is approximately 1.95 seconds when starting 8 meters apart.. Instead of performing an emergency stop, our system can utilize intent prediction to smoothly adjust the robot’s path and avoid potential collisions by predicting into the future.

To evaluate our approach, we modeled a small manufacturing facility where multiple operations occur simultaneously. A human operator attends to various tasks while robots navigate the environment. The operator does not follow predefined or straight paths due to the dynamic nature of the tasks and the need to maneuver around moving robots. This unpredictability, coupled with robot mobility, increases the complexity of perception and intent prediction.

The modeled environment includes four key stations, each representing distinct human actions:

1. 1.

   Storage Area: Retrieving items from storage shelves.

2. 2.

   Workstation: Performing tasks at a workstation.

3. 3.

   Assembly Station: Assembling components.

4. 4.

   Manufacturing Station: Operating manufacturing equipment.

In this setup, the intent prediction problem consists in predicting the workers movement toward one of these stations. By analyzing the human’s trajectory and interaction with the environment, the model infers which station the human is heading toward, thereby determining their intended action.

Figure 2 shows an overview of the system architecture. The system is activated only when a human is detected in the scene. The image processing pipeline extracts objects from the scene, constructs graph structures, and quantifies the robot’s visibility (Section II-C). These graph structures are then processed through a Graph Neural Network (GNN) (Section II-D), generating a prediction for the current time step based on spatial understanding. Additionally, node embeddings from the final GNN layer are extracted and shared with other robots in the system. These embeddings are combined with human pose keypoints from an off-the-shelf object detection algorithm over time to form a sequence. This sequence is passed through a Recurrent Neural Network (RNN), chosen based on whether an ego-centric or collective prediction is required (see Section II-E). The RNN outputs predictions for the current time step and forecasts actions for future timesteps. The predicted output for the current timestep and a confidence value from the calibrated RNN are shared with other robots. This shared information is then processed within a swarm intelligence-inspired consensus mechanism (see Section II-G), where weighted calculations enable the multi-robot system to converge on a unified prediction.

We extract feature vectors for every object mentioned in Table I and the human in the scene using a ResNet50 [30] backbone, which outputs a one-dimensional vector of length 512. For each image, we construct a graph structure where each node represents a concatenation of two feature vectors. The first vector encodes the general appearance of the scene [31]. The second vector captures the local appearance of specific objects found in the image. Inspired by [31], we combine these global and local features by concatenating the two vectors, providing a comprehensive representation for each node in the graph.

To establish a spatial understanding of the scene, we represent the human and surrounding objects as a star-shaped graph, with the human positioned at the center. Each object is connected to the human node with undirected edges weighted by their Euclidean distances (derived from 2D bounding boxes). These edges form an adjacency matrix $A$, which is augmented with self-loops to create a modified adjacency matrix $A^{\prime}=A+I$. The node features, initially $H^{(0)}$, are are iteratively transformed through a 2-layer Graph Convolutional Network (GCN) [32], leveraging the modified adjacency and degree matrices to aggregate information effectively. This shallow 2-layer GCN effectively handles simple star-shaped graphs, minimizing overfitting while ensuring efficient, real-time inference on resource-constrained robotic platforms. Further architectural details are available on the project website.

To grasp the temporal relationship of the scenario, at least two options have been leveraged in the literature: Gated Recurrent Units (GRUs) [33] and Long Short-Term Memory networks (LSTMs) [34]. We select the first due to their lower memory consumption and overall efficiency [35]. We have implemented two instances of GRUs: the ego-GRU and the collective-GRU. The ego-GRU processes information from the ego robot without incorporating data from other robots, while the collective-GRU processes aggregated information from multiple robots.

Implementing these two GRUs allows us to gain insights into the multi-robot system. We can evaluate how a single robot performs in understanding the spatial scene and processing temporal information. Additionally, we can assess the improvement in prediction accuracy when a robot incorporates information from other robots, as well as how much it contributes to the overall prediction accuracy of the system.

The input vectors for the ego-GRU and collective-GRU are illustrated in Figure 3. For the ego-GRU, we concatenate the output of the final layer of the GNN—which is a 1D vector of length 128—with the flattened output of the human keypoints detector (a 1D vector of length 34). This provides a local, ego-centric spatiotemporal prediction.

Messages are used to share node embeddings (outputs of GNNs from other robots) and combined in the collective-GRU model. We use Zenoh [36], which allows robots to share information efficiently and also serves as a middleware for ROS 2. We adopt a straightforward aggregation strategy, averaging all the node embeddings to produce a unified representation. This input format is designed to be both adaptable and scalable, as all inputs are reduced to a fixed-sized vector regardless of the number of participating robots. By leveraging this aggregated data, each robot enhances its predictions with information collected from other robots’ perspectives. However, at this stage, achieving convergence to a single, unified decision about human intent remains unresolved.

To achieve consensus among multiple robots regarding human intent, we implement a consensus algorithm that aggregates individual predictions, weighting them by each robot’s visibility ratio and prediction confidence. Inspired by majority rule mechanisms in collective decision-making [37, 38, 39], this approach is designed to scale with varying numbers of robots.

We define the visibility ratio for each robot $v_{i}=\tfrac{N_{\text{detected},i}}{N_{\text{total}}}$, where $N_{\text{detected},i}$ is the number of detections by robot $i$ and $N_{\text{total}}$ is the total number of detections across all robots. The prediction confidence is defined as $c_{i}=\mathrm{softmax}\bigl{(}z_{i,k}/T\bigr{)}$, where $z_{i,k}$ is the logit score for the top selected class, and $T$ is a temperature parameter that tunes the softmax distribution. Let $M$ be the number of robots. We then normalize these values as $\tilde{v}_{i}=v_{i}/\sum_{r=1}^{M}v_{r}$ and $\tilde{c}_{i}=c_{i}/\sum_{r=1}^{M}c_{r}$, where $v_{r}$ and $c_{r}$ denote the visibility ratio and confidence for each robot $r$, respectively. The weighted vote for each robot is calculated as:

where $\alpha$ and $\beta$ are scalar weights that balance the contributions of visibility and confidence, respectively. A higher visibility ratio $\tilde{v}_{i}$ indicates a more comprehensive view of the scene by robot $i$. Each robot’s predicted action (from a four-class softmax) and its corresponding weighted vote $V_{i}$ are aggregated across all robots. For additional details, please refer to the project website.

We perform simulation-in-the-loop testing, where the starting positions of the robots and the navigation paths differed from those used during training. Ground truth values and predictions were recorded, and the ground truth data was aggregated to match the length of the prediction horizon.

To evaluate the system’s ability to infer human actions under varying conditions, we designed a comprehensive strategy incorporating three distinct scenarios, each progressively increasing in difficulty. These scenarios tested the robustness and adaptability of the system in action inference:

Scenario 1: Ideal Conditions In the first scenario, there are no obstacles present in the environment, and the robots remain stationary. This setting allows the human worker to move in a straight path toward the intended station.

Scenario 2: Static Obstacles Introduced In the second scenario, we introduce static obstacles into the environment while keeping the robots stationary. The presence of obstacles forces the human to navigate around them, resulting in less predictable trajectories.

Scenario 3: Dynamic Environment with Moving Robots The third scenario is the most complex and dynamic. In addition to static obstacles, the robots are also moving within the environment. This creates a highly dynamic setting where both the human and the robots are in motion, and the scene changes continuously.

To further challenge the system, two of the stations were placed in close proximity - details of the environment layout are available on the website - testing the system’s ability to distinguish between similar actions when potential destinations were very close together.

To validate our approach, we compared its performance against two alternative methods inspired by prior work [16]:

Constant Velocity Model (CVM) The CVM is a non-deep learning approach that predicts future states based on simple velocity assumptions. Using an object detection algorithm, we tracked the human’s head to extract keypoints and employed a Kalman filter for temporal tracking. The twelve latest states are used to calculate the average velocity and predict the worker’s goal.

1D Convolutional Neural Network (1D-CNN) The 1D-CNN served as a baseline to evaluate whether simpler models could suffice without requiring graph neural networks (GNNs). Given the one-dimensional nature of the input data, a 1D-CNN aligns well with the task. We implemented a five-layer 1D-CNN architecture inspired by [41], adapting the input and output to suit our application. The node feature of the human was used as input, and the output was integrated with GRUs to function similarly to our spatiotemporal pipeline.

We collected three datasets corresponding to three different simulation scenarios. These datasets were designed to capture the motion, appearance, and state of human operator within the scene from the perspectives of multiple mobile robots positioned at various locations. The datasets consist of temporal sequences; each sequence is a set of frames where the human is either: moving towards a goal, transitioning from one goal to another, and stationary. To ensure effective learning, we maintained an equal distribution of all three cases. Moreover, since we are dealing with a multi-robot system, we extended the datasets to include sequences for all possible combinations involving two robots, three robots, and four robots, respectively. A standard split of 60%/20%/20% was implemented for training, validation, and testing.

Figure 5 confirms the main advantage of our strategy: increasing the number of robots enhances the accuracy of action inference under the same conditions. For these experiments, we used Scenario 3 from our simulation setup (as described in Section III), observed data from the past 2 seconds, and predicted 2 seconds into the future while processing frames at 10Hz.

The results show a significant improvement in accuracy when increasing the number of robots from one to two, and from two to three. However, the improvement is less pronounced when adding a fourth robot. This diminishing return can be attributed to the observation that one of the four robots had a lower-quality view of the scene, resulting in reduced contributions to the collective perception. We hypothesize that further increasing the number of robots would enhance accuracy, as larger swarms tend to exhibit greater fault tolerance. In such cases, the impact of one or even several faulty robots is mitigated by the redundancy and diversity of functioning agents [42].

We present the results for different time horizons in Figure 4, comparing the performance of using one robot versus three robots, with all data processed at 10 Hz. Overall, the results show that having more past information improves prediction accuracy. Specifically, predictions based on 2 seconds (20 frames) of past data outperform those based on 1 second (10 frames). For these cases, predictions extend 10 and 20 frames into the future, respectively, demonstrating better performance with increased historical data aggregation.

When the observation and prediction windows were extended to 3 seconds (30 frames), the system’s performance became inconsistent. While the best-case predictions exceeded those of the 2-second case, the worst-case predictions were significantly worse. This inconsistency arises from the highly dynamic nature of the scene. Accumulating 30 frames increases the likelihood of encountering rapid movements or missed human detections, resulting in sequences of varying quality and more pronounced prediction fluctuations compared to shorter time horizons.

Additionally, processing longer time horizons required more computational resources and introduced greater latency. This highlights a trade-off between achieving higher potential accuracy and maintaining timely responsiveness, depending on the application’s requirements. For our purposes, a 2-second time window offered the optimal balance between prediction accuracy and system responsiveness.

It is crucial to analyze how much temporal information is necessary to optimize computation and memory usage without compromising accuracy. We conducted additional experiments to compare the system’s performance based on the number of frames used for prediction. Processing frames at 10 fps, we examined the effect on accuracy when using all 10 frames, 5 frames, and 3 frames. At runtime, we first accumulate all 10 frames and then pick the required number of frames at uniform intervals. Figure 6 illustrates the results.

Our observations indicate that utilizing all 10 frames leads to better predictions, highlighting the importance of sufficient temporal data for accurate action inference.

Table IV presents the results from Scenario 3 of the simulation, using a 2-second temporal window, and compares them against the CVM and 1D-CNN strategies. While the CVM [43] performs well in Scenario 1, where no obstacles are present and the human moves along a straight path, its accuracy significantly declines in more complex scenarios. Specifically, the CVM struggles to predict human actions when the human changes direction, remains stationary, or stops.

In contrast, our spatiotemporal approach outperforms the 1D-CNN, demonstrating the advantage of integrating spatial and temporal information through GNNs and GRUs. The combination of spatial modeling via GNNs and temporal modeling through RNNs provides a comprehensive understanding of the scene, capturing both relationships between objects and the temporal evolution of actions. This holistic perspective enables our spatiotemporal pipeline to deliver more accurate predictions than methods relying solely on temporal sequences (such as the 1D-CNN) or simplistic motion assumptions (as in the CVM).

Quality of graphs Robots with weaker or sparser graphs—where fewer objects in the scene are linked to the human node—tended to produce less accurate and more unpredictable predictions. The limited number of objects reduces the robot’s ability to model relationships within the scene effectively, leading to decreased prediction performance.

Furthermore, individual robot predictions improved when the graph explicitly linked the source (the human’s current position) to the destination (the intended goal or station). This highlights the importance of graph structures that capture critical relationships between key entities in the environment, as these connections enhance the robot’s spatial understanding.

Erroneous data in small teams In experiments with a relatively small multi-robot system of up to four robots, robots with weaker or sparser graphs not only made poor individual predictions but also negatively influenced the collective perception when their data was aggregated. This suggests that the quality of each robot’s perception plays a significant role in the effectiveness of the consensus mechanism, particularly in smaller teams. These findings align with prior research [42], underscoring the need for robust individual perception to ensure reliable collective decision-making in small multi-robot systems.