SG-LSTM: Social Group LSTM for Robot Navigation
Through Dense Crowds

Research has extensively focused on identifying group features and discovering groups using video input. Notably, [2] adopts YOLOv2 for object detection, converting detections into feature vectors and clustering nearby individuals spatially. Objects are initially considered separate clusters, and groups are formed through iterative processes of computing Euclidean distances and updating centroids. Similarly, [4] defines a social group based on proximity, speed, and movement direction of pedestrians. Additionally, Sathyamoorthy et al. [1] propose mathematical models for pedestrian group characterization, considering walking speed, interaction, group size, and proximity as defining factors.

Apart from physical properties like positions and distances in crowds, group identification can also consider collective behaviors [3]. Shao et al. use the Collective Transition (CT) of pedestrians from tracklets to form clusters, defining groups based on high-velocity correlations with anchor tracklets. This approach provides insights into crowd dynamics and social groupings. These studies highlight diverse strategies, utilizing object detection, feature vectorization, clustering, and collective behavior analysis, to effectively identify and categorize groups in crowds from video data.

This literature review explores a diverse range of research works that contribute to various aspects of efficient and safe navigation in complex scenarios. Cheung et al. [46] propose an innovative vehicle navigation approach based on driver behavior classification, while Chandra et al. [47] introduce CMetric, a driving behavior measure utilizing centrality functions. Randhavane et al. [48] present pedestrian dominance modeling for socially-aware robot navigation, considering pedestrians’ dominance in crowded environments. Bera et al. [49] develop Adapt, a real-time adaptive pedestrian tracking system, and Bera and Manocha [50] present a real-time multilevel crowd-tracking method using reciprocal velocity obstacles. Additionally, Bera et al. [51] introduce GLMP, a real-time pedestrian path prediction model incorporating both global and local movement patterns. Real-time anomaly detection in crowded scenes is addressed by Bera et al. [52], and online parameter learning for data-driven crowd simulation and content generation is proposed by Bera et al. [53]. The research works by Murino et al. [54] explore group and crowd behavior analysis techniques within the domain of computer vision. Furthermore, Bera et al. [55] delve into the concept of “The Socially Invisible Robot,” aiming to enable robots to navigate and interact effectively in social environments. Additionally, Bera et al. [56] present an interactive crowd-behavior learning system facilitating surveillance and training tasks. Chandra et al. [57] propose GraphRQI, a method for classifying driver behaviors using graph spectrums, and Cheung et al. [58] identify driver behaviors using trajectory features to enhance vehicle navigation and safety. These research works collectively contribute to advancing navigation and behavior analysis in various scenarios, providing valuable insights for the development of socially-aware robots and crowd management systems.

The field of computer vision has significantly contributed to modeling pedestrian behavior [5]. The foundational Social Force Model proposed by Helbing and Molnar [5] has found applications in crowd simulation [6] and abnormal behavior detection [9]. Further extensions include joint modeling of pedestrian trajectories and groupings [11] and modeling social and group behavior for multiple people tracking [7]. Notably, there have been studies on people tracking with motion predictions from social forces [8] and detecting social groups in video data [10]. Additionally, image-based motion contexts have been explored for multiple people tracking [12]. Recent works have explored diverse techniques for human activity prediction, including trajectory prediction with PORCA [41], socially-aware navigation using SocioSense [40], and modeling agent interactions with the LSTM-CNN hybrid network [39]. Bera et al. [38] predict pedestrian paths in complex environments. Studies also cover early activity recognition from video streams [31], recognizing human activities from partially observed videos [27], and predicting actions from static scenes [28]. Data-driven activity prediction algorithms have been proposed [29], and Vondrick et al. [30] present a deep learning-based framework for anticipating visual representations. Additionally, the research further explores pedestrian behaviors in stationary crowds, analyzing relationships [17], predicting saliency [18], and socially-aware forecasting [19]. Techniques include data-driven analysis [20], Gaussian process regression flow [21], and trajectory learning [22]. Predictive models cover behavior recognition [23], learning intentions [24], and action forecasting [25]. Kitani et al. [26] forecast future actions from visual input.

Current methods in social navigation often overlook the critical aspect of detecting and preserving the cohesion of social groups in a scene. By neglecting the presence and dynamics of such groups, these approaches may not achieve optimal prediction accuracy and fail to account for the social behavior of individuals. To address this limitation, we present our novel approach, SG-LSTM, which introduces a group-aware paradigm for trajectory prediction. Our approach demonstrates its effectiveness through extensive experiments on various datasets, showcasing its ability to predict socially-compliant trajectories while avoiding collisions with detected groups. By considering group dynamics in trajectory prediction, SG-LSTM represents a significant advancement in the field, promising more reliable and human-friendly robot navigation in complex real-world scenarios.

The SG-LSTM takes raw RGB and Depth frames from videos as input. Our CNN-based group learning algorithm then learns to predict groups in dense crowds, providing spatial coordinates of the groups for the trajectory prediction pipeline. The architecture is visualized in Fig. 3, and all symbols used in this paper are summarized in Tab. I.

Our trajectory prediction algorithm builds upon the Social-LSTM model [32] designed for human trajectory prediction in crowded social scenes. Social-LSTM incorporates social interactions and past trajectories of individuals to predict future movement patterns. Although Social-LSTM is a powerful model for human trajectory prediction in crowded social scenes, it has limitations. Its accuracy tends to decrease with higher pedestrian density or when more agents form groups. By introducing a superior group dynamics model, we enhance the state-of-the-art in two essential ways: achieving more accurate trajectory predictions and better resilience to inaccuracies in detection or sensor output. When we model the group, all pedestrians are considered within one group, allowing the prediction to rely on group trajectories even when tracking for some agents within the group is challenging. This robustness is especially valuable when pedestrian detection algorithms struggle to accurately extract spatial coordinates, particularly for people walking in groups or occluded scenes.

Scaling up Social-LSTM or similar models to handle large and complex scenes with numerous individuals presents challenges due to the substantial computational resources and memory required to capture the social context of the scene. As the model represents each person using an LSTM, a scene with multiple individuals necessitates a corresponding number of LSTMs, leading to increased computational demands. Moreover, these resources may be sub-optimally utilized when dealing with stationary people or individuals moving together as a group. Consequently, there is a growing need for more efficient and scalable trajectory prediction models that can handle such scenarios while maintaining prediction accuracy and computational efficiency.

In our approach, we establish spatial coordinates for each detected group as $(x_{g},y_{g})^{i}_{t}$ at any given time instant $t$. For pedestrians not belonging to any groups, we treat them as individual groups with a single member. These spatial coordinates are collected over five consecutive time steps, from $t_{1}$ to $t_{5}$, at a frame rate of 30 fps, and then fed into our Social-Group-LSTM model to predict trajectories for the subsequent five-time steps, spanning from $t_{6}$ to $t_{10}$. This time-step configuration allows us to effectively compute and forecast the movement patterns of both grouped and individual pedestrians, enhancing trajectory prediction accuracy and enabling socially-compliant navigation in crowded environments.

Our approach involves training a CNN-based group learning model on our annotated dataset. It builds on a single-stage object learning algorithm, utilizing a spatial pyramid pooling (SPP) module to detect objects at various scales and sizes. This capability proves beneficial for identifying groups with diverse shapes and sizes. Additionally, our approach effectively handles occlusions, where objects in a group partially or fully block each other through a combination of spatial features and attention mechanisms, ensuring accurate object detection within groups.

The trained model outputs bounding boxes around groups in crowded scenes. These group bounding boxes are combined with pedestrian bounding boxes for individuals not belonging to any group, which are already available from the dataset. The bounding boxes are then projected onto depth maps for each scene, and the depth of each pedestrian or group is computed using the centroid of the bounding box. By utilizing the pixel depth value ($d$), the frame width, and the camera’s field of view angle ($FOV_{\text{cam}}$) in radians, we calculate spatial coordinates for all pedestrian groups and ungrouped pedestrians. The angular displacement of each pedestrian/group with respect to the robot can be determined using the centroid of the bounding box $(x_{b},y_{b})$ and the FOV of the camera, where $\phi_{g}=({x_{b}}/{w})\cdot FOV_{cam}$. The spatial coordinates of the pedestrians/groups are calculated as follows:$(x_{g},y_{g})=d\cdot[cos\phi_{g},sin\phi_{g}]$.

In our approach, we employ a social pooling layer, similar to Social-LSTM, to analyze group interactions in a scene. The pooling layer is defined by a tensor $H^{i}_{t}$ for the $i^{th}$ trajectory, which is given as:

where $h^{j}_{t-1}$ represents the hidden state corresponding to the $j^{th}$ group at time step $t-1$, and $1_{m,n}$ is an indicator function that checks if the coordinates $(x_{g},y_{g})$ corresponding to the neighboring groups of the $i^{th}$ group, represented as a set $\mathcal{N}_{i}$, are present in the $(m,n)$ cell of the grid. This layer effectively captures the social interactions among all the groups in a scene.

The social pooling layer in the Social-Group LSTM model treats each group as a single entity when modeling pedestrian interactions in a scene. This advantageous approach makes it easier for the model to scale up to denser crowds as not every person in the background is represented by an LSTM, unlike the case in Social-LSTM. Consequently, this significantly reduces the number of parameters required to model a densely-crowded scene. Moreover, by grouping people together, the model overcomes any inaccuracies that arise from computing the spatial coordinates of occluded pedestrians in the background, further enhancing the overall accuracy of trajectory predictions.

Model Layers: Our model layers and weights are similar to the Social-LSTM model. The input coordinates and hidden tensor are embedded into the vector $e^{i}_{t}$ and are passed as input to the hidden state ($h^{i}_{t}$) of an LSTM cell with weight W, corresponding to the current time step of the $i^{th}$ group, $h^{i}_{t}=LSTM(h^{i}_{t-1},e^{i}_{t};W)$.

Trajectory Estimation: The predicted trajectories are assumed to belong to a normal distribution and thus are given as: $(\hat{x_{g}},\hat{y_{g}})^{i}_{t}\sim\mathcal{N}(\mu^{i}_{t},\,\sigma^{i}_{t},\,\rho^{i}_{t})$. Furthermore, the model parameters are trained by minimizing the negative log-Likelihood loss for every $i^{th}$ trajectory.

The trajectories $(x_{g},y_{g})$ returned by the model are passed to a navigation system that can use this to compute optimal paths for a robot. Since we predict trajectories using social interactions between pedestrians employing a group detection model, we can say that the robots could leverage this to deploy a socially compliant navigation system that would avoid obstructing through groups.

We deploy our algorithm on a Unitree robot dog. The algorithm for predicting paths outlined earlier can also be utilized to navigate through dense crowds or pedestrians without collisions. The approach is based on Generalized Velocity Obstacles (GVO) [37]. The combination of path prediction and GVO is used for car-like robots, taking into account their dynamic constraints.

In this context, we are employing kinematic constraints similar to a car and assuming that the robot can detect the location of dynamic obstacles, such as pedestrians and other robots, nearby, despite sensor noise. We have implemented this method on a robot to navigate a crowded walkway toward its intended destination.

The GVO navigation method is a technique that relies on velocity obstacles to navigate robots that have kinematic constraints. In our situation, we employ kinematic constraints similar to those of a car, and we assume that the robot can detect the positions of nearby moving obstacles, such as pedestrians, albeit with some noise. The robot uses our approach to anticipate the movement of each pedestrian as it navigates through the crowded walkway to reach its intended destination.

A conventional kinematic model [37] is utilized for the robot. The robot’s state is characterized by its position $(x_{r},y_{r})$ and orientation $\phi$. It controls the robot’s speed and steering angle, represented by $u_{s}$ and $u_{\phi}$.

The configuration of the robot is expressed as its position $(x_{r},y_{r})$ and orientation $\phi$, and the robot has controls for speed and steering angle, represented by $u_{s}$ and $u_{\phi}$, respectively.

Assuming that the controls remain unchanged for a given time interval, the position of the robot at a specific time $t$ can be determined by the following expression:

The constraints for the robot are specified as follows:

For more details, please refer to [37].

The robot utilizes the prediction scheme we developed to anticipate the paths of the pedestrians and pedestrian groups and prevent any steering actions that could result in a collision. It is assumed that the pedestrians may not actively take measures to avoid colliding with the robot, which implies that the robot is responsible for ensuring collision avoidance.

Our dataset (Fig. 4) is based on data collected on the Purdue Campus at West Lafayette and publicly available 3D point cloud, and RGB-D pedestrian dataset [42]. In this study, we manually annotated perceptually-visible pedestrian groups in crowded scenes from RGB frames. We utilized a reference algorithm to aid in labeling, which involved processing RGB frames with 2D bounding boxes of detected pedestrians and their corresponding depth information. Pedestrians’ positions were extrapolated relative to the robot’s coordinates frame using methods presented in [1]. Additionally, we calculated various features, such as group size, walking speed, and proximity of detected pedestrians, crucial for determining group cohesiveness. To classify pedestrians as a group, we referred to the conditions presented in [1]. Subsequently, for all identified groups, we used CVAT.ai to accurately generate 2D bounding boxes. The labeled data was then used to train our model, following the methods explained in Sec. III. The annotated dataset, which includes defaced, time-synced, color, and depth frames, along with the bounding boxes of pedestrian groups and individual pedestrians, will be released alongside this paper.

By assuming that the pedestrians traveling as a group follow the same trajectory, we do not need to compute the trajectory for every individual pedestrian. Other trajectory-predicting methods that treat pedestrians as individuals would be less efficient in densely crowded environments due to needing to add trajectories for every person in the scene. And when these methods compute the trajectory for one person, it might treat the surrounding people as individual obstacles. In contrast, our model can avoid these expensive computations by knowing that these people belong to the same group. We show that our assumption that a group of pedestrians shares the same trajectory holds by calculating the average displacement error and the final displacement error between the predicted trajectories and the ground truth, where the ground truth is the trajectories of each person in the scene. The computed error in Tab. II confirms that our assumption holds and does not affect the model’s accuracy.

We observe our approach, SG-LSTM, takes, on average, 55.4% less time (Tab. III) than S-LSTM [32]. On average, 30-50% of our tracked pedestrians belong to a group that significantly improves our compute time and is hence more appropriate for edge devices like robots. Lower compute requirements translate to lower costs for both hardware and software development. This means that robot navigation systems can be made more affordable and accessible to a wider range of users. The result is better performance, in terms of accuracy and speed, compared to Social-LSTM.