Jointly Learning Agent and Lane Information for Multimodal Trajectory Prediction

Related agents share the same scenario, co-operating with each other to perform safe actions. Accounting for the social interaction among neighboring agents is critical for safe and reasonable trajectory prediction. Grid-based methods [12, 20] divide the space around agents into 2D grids and apply pooling to aggregate the interaction feature.These methods are hard to represent the irregular real-world interaction. Recently, graph-based methods [22] generalize deep learning on regular grids to arbitrary graphs with irregular topologies. They apply graph convolution [13] or attention [9] to achieve message passing among the vertices composed of agents. These methods have achieved competitive performance in accuracy, however, they consider the social interactions independently, without utilizing the link between the agents and the scene.

Modeling the static scene context is the key to understand the intention and executable actions of the target agent. convolutional neural networks (CNN) based works [5, 6] use CNN to process the rasterized map image to obtain the scene context. These methods neglect the road topology such as the neighbor but opposite lanes. Recent methods use connective lanes sampled from the map as scene data. graph nerual networks (GNN) based methods[3, 15, 2] take the sampled points of lane centerline from the semantic map as vertices on a graph and obtain the scene context through GNN. They achieve competitive distance-based metrics but poor multimodal. lane-based methods[7, 10, 9] respectively propose a lane attention mechanism to use lane as scene data. MTPLA[7] utilizes the most correlated lane. LaPred[10] integrates all of the lane feature with learnable weights. WIMP[9] focus on the effective segments of the most correlated lane. We share a similar idea to lane-based methods. The difference is that we jointly represent each lane with their associated agents and separately treat each lane by proposed RLA.

The future motion of agents is inherently multimodal, it requires to generate multiple trajectories with likelihoods. Some works are built upon stochastic models such as generative adversarial networks (GANs) [16] or variational autoencoders (VAEs) [17]. Despite their competitive performance, the drawback of requiring multiple sampling to latent variables during inference prevents them from being applied. To avoid mode collapse [4], recent frameworks decompose the task into classification over anchor [14] or goal [18], followed by conditional regression. TNT [14] take the lane centerline nodes as anchors which are accurate but too massive. GoalNet [18] propose multimodal generators to select goals among lanes but it mainly use rasterized map. Generally, lane information offers a strong prior on the semantic behavior of drivers. Driven by this, we take all possible lanes as goals and predict based on them.

The purpose of trajectory prediction is to predict the future motion of all agents in a scene, given their past motion and current scene data. Since we focus on the multimodal problem, we only consider the one agent we called target in a scene. Suppose that the target agent has $N$ possible goals $G=\{{g^{i}},i\in[1,N]\}$ on its situation, considering that drivers usually choose a potential goal $g^{i}$ to drive in reality and different goals can be treated independently, the task can be formulated as finding the posterior distribution of the future trajectory $Y$ given the possible goals $G$:

with approximating the distribution of ${\rm{Y}}$ as $K$ representative future trajectories ${\rm{Y}}=\left\{{{Y^{k}},k\in[1,K]}\right\}$, where ${Y^{k}}=\{y_{t}^{k},t\in[1,{T_{f}}]\}$ denotes the $k$-th future trajectory over future time horizon ${T_{f}}$ and $y_{t}\in\mathbb{R}^{2}$ denotes the agent’s 2D coordinates in bird’s eye view(BEV) at time $t$.

The possible goals $G$ depend on the history motion state of the target agent, the distribution of nearby dynamic agents and static scene which are essentially drivable paths. The motion state of target agent and $M$ surrounding agents we called the social agents over the past ${T_{P}}$ timestamp can be formulated as ${\rm{X}}=\left\{{{x_{t}},t\in[-{T_{p}}+1,0]}\right\}$ and ${\rm{S}}=\left\{{{S^{j}},j\in[1,M]}\right\}$ respectively, where ${S^{j}}=\left\{{s_{t}^{j},t\in[-{T_{p}}+1,0]}\right\}$ denotes the past trajectory of $j$-th social agent and $x_{t},s_{t}\in\mathbb{R}^{2}$ denote the agents’ BEV coordinates at time $t$. Similar to [9, 10], we describe the drivable paths as the concatenated centerline of lane segments, which called lane proposals corresponding to the $N$ goals and denoted as ${\rm{L}}=\left\{{{L^{i}},i\in[1,N]}\right\}$, with ${L^{i}}=\left\{{l_{h}^{i},h\in[1,H]}\right\}$ and $l_{h}^{i}\in\mathbb{R}^{2}$ representing $h$-th lane node of $i$-th lane because the path are composed of discrete samples, $H$ is the length of a lane.

As mentioned above, $G$ boll down to the sum of ${\rm{X}}$ , ${\rm{S}}$ and ${\rm{L}}$. In most cases the drivers tend to choose a lane proposal as their goal and drive along the centerline of the lane proposal. In addition, ${\rm{S}}$ and ${\rm{L}}$ are interrelated and mutually influenced, which can be specifically performed the distribution of surrounding dynamic agents will simultaneously affect the drivability of lane proposals. Therefore, we comprehensively aggregate the social agents’ situation ${\rm{S}}$ into lane proposals ${\rm{L}}$ to obtain instance-level lanes ${\rm{\tilde{L}}}=\left\{{{{\tilde{L}}^{i}},i\in[1,N]}\right\}$, then we can regard the $N$ goals as the evolution results of the current motion state ${\rm{X}}$ on $N$ instance-level lanes, so (1) could be expressed as

where ${\tilde{L}^{i}}$ represent $i$-th instance-level lane under ${g^{i}}$, we can adequately characterize the distribution of future trajectories by aggregating the trajectory predicted under $N$ instance-level lanes. We consider the multimodal trajectory prediction problem and build our model driven by Eq. (2).

The model structure of the proposed network is corresponding to the formulation expressed in III-A. Fig. 1 depicts the overall structure of our network. The lane proposals obtaind by preprocessing and the social agents’ past trajectories are respectively fed into the multi-scale one-dimensional Convolutional Neural Network (1D-CNN) and the 1D-CNN followed by Gated Recurrent Unit (GRU) to encode as the lane proposals feature ${\rm{L}}$ and social motion feature ${\rm{S}}$. For effectively representing the road information as instance-level lane ${\tilde{\rm{L}}}$, we apply a spatial attention in S2L to jointly fusing the ${\rm{L}}$ and ${\rm{S}}$. The instance-level lane will serve as the extra input to RLA modules. For the target agent, by applying the 1DCNN followed by RLA Encoder, we can encode the past motion feature ${\rm{X}}$ and current lane condition ${\tilde{L}^{i}}$ to hidden vector $h^{i}$. By feeding the vector to Lane Selector, we can get the lane score which marked as the transparency. Finally, based on the hidden vector and lane score, the RLA Decoder and Trajectory Selector will produce and select K typical trajectories with probabilities.

The regression and two classification tasks are independent and the whole model can be trained end-to-end with a total loss function:

where ${\lambda_{1}}$ and ${\lambda_{2}}$ are hyperparameters to balance the tasks.

For regression, we apply the ${l_{1}}$ loss on all predicted time steps that proved effective in previous works [9]:

where $\hat{y}_{t}^{k}$ denotes the future position of the k-th trajectory at time t and ${y_{t}}$ is the corresponding ground truth.

For classification, we use the cross-entropy loss between the output scores and the labels:

where the labels are generated during training with self-supervised learning task by

with $D_{1}\left(L,Y_{GT}\right)=\sum_{t=1}^{T_{f}}\beta(t)\min_{h\in\{1,\ldots,H\}}\left\|y_{t}-l_{h}\right\|$ and $D_{2}\left(\mathrm{Y},\mathrm{Y}_{GT}\right)=\sum_{t=1}^{T_{f}}\left\|y_{t}-\hat{y}_{t}\right\|$ denoting the distance between the ground truth trajectory and the lane, the prediction respectively, where $\beta(t)=t$ is the scaling weight to indicate the importance of different time steps. Note that there is only one ground truth trajectory, in order to make the model better learn to follow a certain lane, we only predict the trajectory under the most likely lane during training, when inference, we input all of the lane proposal and apply the selectors to get final trajectories.

We compare JAL-MTP with the existing methods that have similar configuration and SOTA performance. Table. I presents the results on the validation set with $K=1,6$. GRU ED is the baseline that simply GRU encoder-decoder structure. MTPLA [7], WIMP [9] and LaPred [10] are lane-based methods that also use instance-level lane similar to us, the difference is that MTPLA and LaPred map the entire lane to a vector, WIMP consider without the explicit changes in position and curvature of the lane segment. LaneGCN [3] and TPCN [15] are the GNN-based models that have better performance in accurate metrics through GNN but perform bad in multimodal results. The quantitative results show that our method out-performs all related lane-based works in almost all metrics and achieves similar performance to the GNN-based models. Notably, our ${\rm{minFD}}{{\rm{E}}_{6}}$ and MR metric are the best among all baselines, which indicates that our method has a competitive ability to capture the main mode while considering the multimodal of predicted trajectories.

Fig. 6 shows the qualitative results of our model to other competitive methods on particular hard scenarios selected from Argoverse dataset. ${\rm{Ours\_all}}$ visualizes all trajectories generated by our model for all reachable lanes and ${\rm{Ours\_6}}$ shows the remaining trajectories filtered by two selectors. Our model could generate trajectories covering all plausible modes in each challenging scenario. The trajectories with reasonable probabilities produced by ${\rm{Ours\_6}}$ show the importance of two selectors to identify typical trajectories.

Specifically from Fig. 6, in turning and forking cases, it is hard to distinguish whether the agent will go on or turn to another path, the baselines only maintain current motion but our model can suggest both paths even there is only a small possibility. In intersection scenario, although there are more possible paths agent can choose, our model can still explicitly predict map-adaptive trajectories compared with others. Moreover, when there is congestion at the front, only our model correctly predicts the deceleration and tend to bypass. It reveals the effectiveness of jointly learning the social agents and lanes information to help prediction more reasonable and safe.