Maneuver-Aware Pooling for Vehicle Trajectory Prediction

Recurrent Neural Networks (RNNs) represent a rich class of dynamic models that extend feedforward networks for sequence generation in diverse domains like speech recognition [9], machine translation [10] and image captioning [11]. Long-Short Term Memory (LSTM) is an instance of RNNs that has been proved successful in sequence learning and generation tasks [12, 4, 11]. Since motion prediction can be considered as a sequence generation task, and inspired by the LSTM success in this domain, a number of RNN-based approaches have been proposed for trajectory prediction [4, 2, 3, 6, 13, 14, 15] of both pedestrians and vehicles.

In their seminal work Social LSTM, Alahi et al. [4] extended RNNs for human trajectory prediction using a social pooling layer that models nearby pedestrians. Gupta et al. proposed a Generative Adversarial Networks [16] (GAN): a RNN Encoder-Decoder generator and a RNN based encoder discriminator, to predict socially-acceptable multimodal pedestrian trajectories [2]. Social LSTM approach was further improved in [3] by using convolutional social pooling applied to vehicle motion prediction on highways. LSTM network is also used to predict the location of vehicles in an occupancy grid [17] at different future intervals. Convolutional networks and LSTM were combined to predict multi-modal trajectories for an agent on a bird’s eye view image [18].

Deo et al. proposed a model [3] that outputs a multi-modal predictive distribution over future trajectories based on three lateral and two longitudinal maneuver classes. MultiPath model [1] can predict a discrete distribution over a set of future state-sequence anchors and output multi-modal future distributions. The GAN based encoder-decoder architecture in [2] encourages diverse multimodal predictions of pedestrian trajectories with the introduced variety loss. A multi-modal probabilistic prediction approach was presented in [5] based on a Conditional Variational Autoencoder and is capable of jointly predicting sequential motions of each pair of interacting vehicles. Zyner et al. [6] presented a model based on RNN with a mixture density network output layer, for predicting driver intent at urban single-lane roundabouts through multi-modal trajectory prediction with uncertainty.

Social LSTM (S-LSTM) has been introduced for pedestrian motion prediction [4], where one LSTM is assigned to each person in a scene. To capture the interaction of people in a neighborhood, neighboring LSTMs are connected through a “Social” tensor. This tensor is constructed by defining a spatial grid around the agent being predicted (Fig. 1 left), and populating the grid with LSTM states of this agent and its neighbors based on their spatial configuration. A fully connected and a SUM pooling layers are then applied to the social tensor to produce the pooling vector.

Deo et al. introduced convolutional social pooling (CSP) [3] to predict vehicle motion behaviors on highways. They extended the work of [4] improving the pooling layer by using convolutional and Max-pooling (rather-than a fully-connected and SUM-pooling) layers to embed the LSTM states populated in the social tensor.

In their Social-GAN (S-GAN) work [2], Gupta et al. argued that the grid-based pooling scheme presented in Social Pooling [4, 3] is a hand-crafted solution that is slow and fails to capture global context. They proposed to concatenate the relative position between an agent and all its neighbors (Fig. 1 center) with the LSTM state of each agent. The concatenated tensor is then processed independently by a Multi-Layer Perceptron and a Max-pooling layer.

The previous approaches depend solely on the lateral and longitudinal positions to represent the vehicle trajectories. In contrast, we employ polar coordinates, vehicles orientation and radial motion velocity to represent the vehicle trajectories. Using such higher-order information results in an implicitly maneuver-aware pooling strategy, that proved to outperform the state-of-art pooling approaches in terms of the prediction accuracy.

The inputs to our model are track histories:

$\displaystyle\begin{split}X=[x^{(t-t_{h})},...,x^{(t-1)},x^{(t)}]\end{split}$

(1)

where $t_{h}$ is a fixed (history) time horizon, and at any time instant $t$,

$\displaystyle\begin{split}x^{(t)}=[r_{0}^{(t)},\phi_{0}^{(t)},V_{r_{0}}^{(t)},r_{1}^{(t)},\phi_{1}^{(t)},V_{r_{1}}^{(t)},...,r_{n}^{(t)},\phi_{n}^{(t)},V_{r_{n}}^{(t)}]\end{split}$

(2)

represents the position (represented by a distance $r$ and an angle $\phi$) and radial velocity $V_{r}$ of the ego vehicle (subscript $0$), and the neighbor vehicles (subscripts $1$ to $n$), represented in polar coordinates.

The output of the model is a probability distribution over:

$\displaystyle\begin{split}Y=[y^{(t+1)},...,y^{(t+t_{f})}]\end{split}$

(3)

where $t_{f}$ is a fixed (future) time horizon, and at any time instant $t$,

$\displaystyle\begin{split}y^{(t)}=[r_{0}^{(t)},\phi_{0}^{(t)},V_{r_{0}}^{(t)}]\end{split}$

(4)

represents the future position and velocity of the ego vehicle being predicted.

The multi-modal distribution over future trajectories can be hierarchically factorized [3, 1]. First, estimate the intent uncertainty over a set of maneuver classes. Second, given the intended maneuvers, predict the vehicle future trajectory.

The intentions of human drivers are modelled using two discrete sets of maneuver types: (1) location-wise maneuvers $M_{l}=\{m_{l_{p}}\}_{p=1}^{P}$, and (2) acceleration-wise maneuvers $M_{a}=\{m_{a_{q}}\}_{q=1}^{Q}$. We model the uncertainty over each discrete set of maneuvers using a softmax distribution.

Given the intended maneuvers $m$, the uncertainty over the future trajectory is modelled as a Gaussian distribution:

$\displaystyle\begin{split}P_{\Theta}(Y|m)=N(Y|\mu(X),\Sigma(X))\end{split}$

(5)

that is parameterized by [3]:

$\displaystyle\begin{split}\Theta=[\Theta^{(t+1)},...,\Theta^{(t+t_{f})}]\end{split}$

(6)

which are the parameters of a multivariate Gaussian distribution at each time step in the future. At any time $t$, this is given by the Gaussian parameters: $\Theta^{t}=[\mu^{(t)},\Sigma^{(t)}]$, corresponding to the mean and variance of the future vehicle position and velocity.

The multi-modal output conditional distribution over future trajectories can now be expanded in terms of the maneuvers:

$\displaystyle\begin{split}P(Y|X)=\sum P(m|X)P_{\Theta}(Y|m,X)\end{split}$

(7)

which yields a Gaussian Mixture Model distribution (GMM) [1]. The mixture weights are defined by the probabilities of the maneuvers.

We encode the state of motion of each vehicle using an LSTM encoder. At any time $t$, a sequence of $t_{h}$ time steps of the track history is passed through the encoder. The LSTM states for each vehicle are updated frame by frame over the $t_{h}$ past frames [3]. The LSTM weights are shared across the sequences of all vehicles [4].

The interaction of vehicles in a given scene is captured by the pooling module. This is achieved by pooling the LSTM states of the neighbor vehicles around the ego vehicle. The output is a pooling vector summarizing the information needed by the ego vehicle to make a maneuver decision. We extend the pooling mechanism in [2] to implement our pooling strategy detailed in Sec. IV.

First, at any time step $t$ the neighbor vehicles around the ego vehicle are identified. The relative position and radial velocity of the ego vehicle and between each neighbor and the ego are then computed (relative to the position and velocity at the origin $O$. Second, the relative position and velocity are concatenated with each vehicle’s LSTM hidden state, and processed independently by a Multi-Layer Perceptron (MLP). Finally, the MLP output is Max-pooled element-wise to compute the pooling vector of the ego vehicle. This method can capture the inter-dependencies of the motion of neighbor vehicles, without being restricted to a specific grid size. More importantly, being based on vehicle orientation and radial motion velocity, the pooling operation becomes more aware of the intended maneuvers.

The location-wise $M_{l}$ maneuvers (Sec. III-B) include three classes: lane keeping, left lane-change and right lane-change, which correspond to semantic location-change concepts on highways. Similarly, the acceleration-wise $M_{a}$ maneuvers include: constant speed, speeding up and slowing down classes. A vehicle is annotated with one of the lane-change classes if it changes its lane during the prediction horizon, while marked under the lane-keeping class otherwise. The NGSIM dataset provides the acceleration of each vehicle. A vehicle is annotated with a slowing/speeding class if its mean future acceleration is less/larger than $-a$/$+a$ threshold, while marked under the constant-speed class otherwise.

The maneuver recognition module consists of two softmax layers to recognize the location and acceleration classes. The input to this module is the LSTM state of the ego vehicle augmented by the pooling vector from the pooling module. Each softmax layer outputs the probability of each maneuver class. The maneuver encoding block in Fig. 2 encodes the output from each softmax layer into a one-hot vector. Both one-hot vectors are concatenated with the trajectory encoding and the resulting tensor is passed to the decoder module.

An LSTM decoder is used to generate the conditional distribution over future trajectories of the ego vehicle. At any time $t$, The decoder generates the future trajectories over the next $t_{f}$ time steps. At each time step, the decoder outputs a multi-modal distribution conditioned on the maneuver classes.

The NGSIM public dataset is used for our experiments. This dataset consists of two subsets: US-101 [7] and I-80 [8] of trajectories of real freeway traffic captured at 10 Hz. Each subset consists of three 15-min segments recorded over a time span of 45 minutes. The recorded segments represent mild, moderate and congested traffic conditions.

The dataset provides the longitudinal and lateral co-ordinates of vehicles projected to a local co-ordinate system, in addition to other data like velocity and acceleration. We divide the complete NGSIM dataset into training $(72\%)$, validation $(10\%)$ and testing $(18\%)$ sets. All sets are randomly sampled from both US-101 and I-80 subsets.

Vehicle trajectories are split into segments of 8 s, where we use $t_{h}=3$ s of track history and a $t_{h}=5$ s prediction horizon. These 8 s segments are sampled at the dataset sampling rate of 10 Hz, and then downsampled by a factor of 2 before feeding them to the LSTMs, to reduce the model complexity [3].

We extended the model in [3] to: (1) incorporate our benchmark pooling strategies, and (2) use multi-variate Gaussian distribution. The SUM pooling layer used in S-LSTM [4] is implemented using a two-dimensional kernel of size (4,3). The sizes of the convolutional social pooling layers of CSP are set as given in [3]. The MLP used in the proposed pooling module (and in S-GAN pooling [2]) has a size of 256 and is followed by a leaky-ReLU layer. The acceleration threshold in Sec.V-C is set to $a=0.2m/s^{2}$. The spatial grid in Fig. 1 is set to a 13 × 3 size, where each column corresponds to a single lane, and the rows are separated by a distance of 15 feet which approximately equals one car length [3]. To emphasize the effectiveness of the proposed pooling strategy, we set the distances $l_{lat}$ and $l_{long}$ to the same width and length, respectively of the spatial grid. The model is implemented using PyTorch [19] and trained end-to-end.

All results are reported in terms of the root of the mean squared error (RMSE) of the predicted trajectories with respect to the ground truth future trajectories, over the prediction horizon. Since the LSTM models generate bi-variate Gaussian distributions, the means of the Gaussian components are used for RMSE calculation.

The results of the following pooling approaches are compared:

• •

  S-LSTM: Social Pooling [4].

• •

  CSP: Convolutional Social Pooling [3].

• •

  S-GAN: the pooling strategy used in S-GAN [2].

• •

  Polar: the proposed pooling strategy where trajectories are modelled by position only in the polar coordinates $r$ and $\phi$, and the model outputs a bi-variate Gaussian distribution.

• •

  Polar-$\bm{V_{r}}$: the full proposed pooling strategy presenting trajectories by position and radial velocity in polar coordinates ($r,\phi,V_{r}$), and the model outputs a multi-variate Gaussian distribution.

While we employ polar representation of vehicle trajectories, other baselines (S-LSTM, CSP, S-GAN) use Euclidean lateral and longitudinal positions. The same model and testing data are used to evaluate every pooling strategy. The maneuver recognition module (Sec. V-C) was incorporated in the Polar-$\bm{V_{r}}$ model only. At the evaluation time, this model outputs the maximum a posteriori probability (MAP) trajectory estimate corresponding to the maneuver classes having the maximum probability.

Besides reporting the overall prediction accuracy, we evaluate the baselines on the following maneuvers: (1) car-following by keeping the same lane (keep), (2) merging from the onramp lane (merge), (3) left lane-change (left), and (4) right lane-change (right). The overall performance of the baseline approaches in Table I implies two findings¹¹1We also noted the same findings for the maneuver-based evaluation.: the baselines have comparable prediction accuracy, and RMSE accuracy was not improved by using the maneuver recognition module. While the former motivates for a different direction of improvement (other than the pooling layer type and the neighborhood structure), the latter suggests feeding higher-order information to the pooling operation.

RMSE results of the maneuver-based evaluation are reported in Tables II (overall and keep) and III (merge, left and right). We note that the overall prediction accuracy is dominated by the keep maneuver accuracy. This is due to the inherent biasness in the data: as drivers tend to keep their lanes most of the time, the dataset provides fewer lane-change examples. However, Our proposed solution tackled this problem without changing the dataset (e.g. by augmentation).

Representing trajectories using relative distance and angle (Polar) is shown to outperform the Euclidean representation adopted by the state-of-the-art strategies, especially at the lane-change and merging maneuvers. The main reason of this improvement is employing the vehicle orientation which makes the pooling operation more maneuver-aware and hence the learning process more efficient.

The maneuver-recognition module estimates the location and acceleration maneuver classes. The input to this module is the embedding of the vehicle dynamics encoded by the LSTM hidden states. It is reasonable to expect that a single-layer LSTM can encode the change in vehicle location into states that are informative for estimating the change in location. However, the accuracy of such networks degrades when encoding the higher-order dynamics necessary for estimating the acceleration. Therefore, we augment the trajectory representation by the vehicle velocity. More importantly and inline with our proposed polar representation, we employ radial motion velocity that encodes vehicle orientation. This defines our full proposed model Polar-$\bm{V_{r}}$ that further improved the prediction accuracy.