The Importance of Prior Knowledge in Precise Multimodal Prediction

Self-driving vehicles (SDV) have the potential to make a large impact on our society, making transportation safer, cheaper and more efficient. A core component of every self-driving vehicle is its ability to perceive the world (including dynamic objects), and forecast how the future might unroll. In recent years there has been incredible progress in perception systems [1, 2, 3]. Many challenges still remain, however, in providing motion forecasts that are simultaneously diverse and precise [4]. That is, having the ability to cover all the modes of the data distribution while generating few highly unrealistic trajectories.

Roads in modern cities have well defined geometries, topologies, as well as traffic rules. The vast majority of actors in the scene will adhere to this structure, for example driving close to the middle of their lane, respecting stop signs, or obeying yielding laws. These agents will also most likely act in a socially acceptable manner, avoiding collisions with other traffic participants. Despite this fact, most perception and motion forecasting systems are trained to be as close as possible to the ground truth employing symmetric loss functions that do not take this structure into account. For example, euclidean distance at the waypoint level between the predicted and ground-truth future trajectories is a common choice for motion forecasting. This can cause uncomfortable rides for the self-driving vehicle with plenty of sudden brakes and steering changes due to false positive motion forecasts intruding into the ego-car’s lane, such as the red trajectory illustrated in Fig. 1-Left. Even worse, these sudden reactions to avoid an imminent collision with a motion forecast can cause an actual collision (with the ground-truth) as a by-product. It is also critical to recall all actors (and their future motion) in the SDV lane, since otherwise it would look like free space ahead and the SDV could dangerously accelerate, as depicted by the red motion forecast in Fig. 1-Right.

One possible solution is to add the aforementioned intuitions into the motion forecasting model as hard constraints. Unfortunately, this may not be resilient to non-compliant behavior from other actors as well as map failures, predicting possibly unrealistic and dangerous situations. In this paper we take an alternative approach and design loss functions that encourage our perception and prediction system to only violate these constraints when they happen in reality.

Incorporating prior knowledge via loss functions is easy when the perception and prediction modules are deterministic. However, deterministic systems fail to capture the inherent uncertainty of the future. This can be catastrophic when it fails to capture the true actor intention (e.g., crossing the street vs waiting, yielding vs not). In order to plan a safe maneuver, coverage of the possible future scenarios is required, along with information about the likelihood of each possible future such that the motion planner can choose the plan with the lowest expected cost. The Gaussian distribution and mixtures thereof have been widely used to represent uncertainty over spatial locations [5, 6, 7]. However, as shown in [4], maximizing the log likelihood of data encourages the model to produce distributions with high recall in order to avoid the big penalty associated with low-density areas. As a consequence many unrealistic samples are generated, sacrificing the precision of the model.

In this paper, we show that making explicit use of our prior knowledge about the geometry and topology of the roads as well as the traffic rules can provide more precise distributions over future outcomes while preserving their recall. However, this is challenging as these priors are typically non-differentiable and thus not directly amenable to gradient based optimization. For instance, the fact that humans tend to follow traffic rules can be better described as a discrete (follow / not follow) action. To this end, we propose a flexible framework to incorporate non-differentiable prior knowledge as a loss and exploit the popular REINFORCE [8] gradient estimator. Our formulation allows us to optimize for any prior knowledge on future trajectories, as long as drawing samples from the perception and prediction model, and obtaining their likelihood can be done efficiently. In particular, we apply our formulation to model how the vehicles interact with the map, encouraging the predictions to respect lane dividers and traffic lights. We also exploit our framework to make the motion forecasting module more planning aware by emphasizing the importance of high recall and high precision near the SDV route.

Our experiments show that our proposed framework can improve the map understanding of state-of-the-art motion forecasting methods in very complex, partially observable urban environments in two challenging real-world datasets: ATG4D [2] and nuScenes [9]. Importantly, our approach achieves significant improvements in the precision of the trajectory distribution, while maintaining the recall. Unlike previous works, in this paper we advocate for measuring the system-level impact of the motion forecasts via motion planning metrics. We demonstrate that including prior knowledge not only results in more comfortable rides, but also in major safety improvements over the decisions taken by a state-of-the-art motion planner [10]. Moreover, we show that achieving a lower minADE alone, the most frequently used metric to benchmark multi-modal motion forecasts, may not translate into safer motion plans.

In order to plan a safe maneuver, we must effectively deal with noisy and partial observations from sensor data, and provide an accurate characterization of the distribution over future trajectories of all actors in the scene. We thus want to model $p(Y|X)$, with $X=\begin{Bmatrix}x_{1},x_{2},\cdots,x_{N}\end{Bmatrix}$ the observations (i.e. a local context for each actor) and $Y=\begin{Bmatrix}y_{1},y_{2},\cdots,y_{N}\end{Bmatrix}$ the future trajectories of all $N$ actors. This is particularly challenging since the future is inherently uncertain, and actors’ discrete decisions induce highly multimodal distributions (e.g., turning right vs. going straight at an intersection, going vs. stopping at a yield sign).

In traditional self-driving stacks, there is an object detection module responsible for recognizing traffic participants in the scene, followed by a motion forecasting module that predicts how the scene might unroll, given the current state of each actor. However, the actor state is typically a very compact representation including pose, velocity, and acceleration. As a consequence, it is hard to incorporate uncertainty coming from sources such as sensor noise and occlusion.

FAF [11] unified these two tasks by having a single, fully convolutional backbone network predict both the current and future states for each pixel in a bird’s eye view grid, directly from a voxelized LiDAR pointcloud. This naturally propagates uncertainty between the two tasks in the feature space, without any need for handcrafted, intermediate representations. [12] extended this framework to include the map as an input by adding a parallel fully convolutional backbone network to process a semantic raster map of the scene, thus making fusion trivial by concatenation. Recently, this framework was further extended to model agent-agent interactions via graph neural networks [5], learn a cost map for motion planning [13], and add differentiable tracking in-the-loop [14]. While these works are great at dealing with uncertainties at the sensor level and mitigating failures downstream from object detection uncertainty by learning joint features, they do not focus on the output parameterization, and all of them produce uni-modal predictions. Uni-modal predictions can result in unsafe behaviors if, for example, the predicted intention is not accurate (e.g., a pedestrian crossing vs waiting) or the predictions lie inbetween two modes (e.g. at branching roads). In this work, we extend this framework to predict multi-modal behaviors that are aligned with human prior knowledge.

The motion planning algorithms in an SDV need to take into account all possibilities to make safe decisions. Thus, motion forecasting models that can characterize complex multi-modal distributions are a must, and efficient sampling is desired such that motion planning can timely find the plan with the lowest expected cost. Approaches that directly output the parameters of the marginal distribution at each timestep with a closed-form likelihood such as a sequence of Gaussians over time [15, 5] meet the efficient sampling requirement, but can suffer from low expressivity. [6] proposed a more stable way to train mixtures of future trajectories than directly optimizing the likelihood, by only training the Gaussian likelihood of the closest mode to the ground-truth trajectory and applying cross-entropy to the mode probabilities. [7] followed up on this idea by anchoring the predictions to clusters produced offline by running k-means on the training set. These two models have a good tradeoff between sampling efficiency and expressivity. Predicting discrete occupancy maps into the future [16] has very good expressivity and naturally captures multi-modality, but is very memory intensive and requires adaptations in order to use traditional motion planners designed for trajectory representations.

Autoregressive models [4, 17, 18] require sequential sampling and thus are not amenable to real-time inference, which is necessary in safety critical applications such as self-driving. Furthermore, there is a mismatch between training and inference: the distributions at training are conditioned on ground truth, but during inference they are conditioned on model samples. This makes the model less robust to noise, particularly in the joint detection and prediction setting where perfect detection is not possible. Finally, modeling interactions between the traffic participants [15, 19, 5, 17, 18] has been shown to help reduce the prediction uncertainty, and generate more socially consistent predictions.

There has been incredible progress in learning multimodal futures with a likelihood objective without leveraging prior knowledge or known structure, thanks to the availability of large and diverse driving datasets. However, datasets collected from real-world driving can only cover one possible mode out of many possibilities of how the future might have unrolled. As shown in [4], this partial description of the underlying distribution along with a Maximum Likelihood Estimation (MLE) objective results in high entropy distributions that favors mode-covering over mode-seeking behaviors [20]. Empirically, although the learned distribution recovers the ground truth future, it also generates highly implausible samples (e.g., out-of-map predictions). In this paper we show that coverage at the expense of precision severely harms motion planning.

A solution to avoid mode-covering predictions is to directly leverage prior knowledge to characterize the true distribution. In particular, the actor’s motion history, road geometry, traffic rules, and nearby traffic participants all constrain the space of plausible futures. Past approaches have (i) devised neural networks with inductive priors at the model level [21], and (ii) use a map-based reconstruction loss as an auxiliary task (e.g. recovering the road mask shape [22] to encourage prediction to fall on drivable areas, which they show helps with generalization to novel driving scenarios). However, their approach is limited since the policy is non probabilistic, and the loss is applied to only the maximum a posteriori (MAP) estimate. In this paper, we propose a more general and powerful approach by leveraging the REINFORCE gradient estimator to incorporate any prior knowledge (including non-differentiable functions) over probabilistic motion forecasts, still allowing the model to recover non-compliant behavior.

In this section, we describe a novel framework to incorporate prior knowledge explicitly into probabilistic motion forecasts. Importantly, our approach still permits the model to predict non-compliant behavior that does not follow traffic rules in the rare event that this occurs. Our method is general and can be applied to any model that can generate actor trajectory samples $y_{n}$ and evaluate their marginal likelihood $p(y_{n}|x_{n})$ efficiently. Since for the remainder of the paper we only refer to per-actor marginal likelihoods, we simplify the notation and refer to an actor’s trajectory as $y$, and its local context as $x$ (i.e. the detected bounding box and local LiDAR/map features). We defer the explanation of the particular state-of-the-art perception and prediction model we use to Section IV, as this is not the main contribution of our work.

Given a traffic scene, humans have rich prior knowledge over how the traffic participants might behave. In this paper, we propose to directly use this prior knowledge as supervision when learning an actor’s distribution over future trajectories. Towards this goal, we encode the prior knowledge as a deterministic reward function $r(y,x)$. We then define the prior knowledge objective as the negative expected reward over samples from the future trajectory distribution. Note that applying the loss directly to the point estimate of the means is not sufficient, since our goal is to learn an accurate characterization of the full distribution for safe motion planning. The goal is then to learn a stochastic policy (parameterized by $\theta$) that maximizes the expected reward:

	$\displaystyle\mathcal{L}_{\text{prior}}(x;\theta)$	$\displaystyle=\mathop{\mathbb{E}_{y\sim p_{\theta}(y|x)}}\left[-r\left(y,x)\right)\right]$
		$\displaystyle=\int-p_{\theta}\left(y|x\right)r\left(y,x\right)dy$

Most priors are non-differentiable and cannot be easily relaxed (e.g., motion forecast following the traffic rules or not) Thus, we leverage policy gradient algorithms, which do not assume differentiability of the reward function $r$ and allow direct optimization without making any approximations. In particular, we use the popular REINFORCE algorithm [8], which only requires the policy to be differentiable, provide efficient sampling, and allow likelihood evaluation. In this case, the gradient of the expected loss can be computed as:

$\displaystyle\nabla\mathcal{L}_{\text{prior}}(x;\theta)=\mathop{\mathbb{E}_{y\sim p_{\theta}\left(y|x\right)}}\left[-\nabla\log p_{\theta}\left(y|x\right)r\left(y,x\right)\right]$

The expectation can then be approximated by drawing samples from the predicted distribution as follows:

$$\nabla\mathcal{L}_{\text{prior}}(x;\theta)\approx\frac{1}{S}\sum_{i}^{S}-\nabla\log p_{\theta}\left(y^{i}|x\right)r(y^{i},x)$$

where $y^{i}$ is the $i$th trajectory sample, over $S$ samples. Although this Monte Carlo estimation is unbiased, it typically has high variance. Our experiments show that this does not pose a problem when using a policy that has an efficient sampling mechanism, since we can draw a large number of samples.

In practice, our reward explicitly incorporates the prior knowledge that drivers generally follow the reachable lanes defined by lane markers, traffic signs and traffic lights. Furthermore, we leverage our knowledge about the self driving vehicle’s planned route to place emphasis on the most relevant actors. Intuitively, missing the prediction of an actor coming in conflict with the SDV route (false negatives) or predicting that an actor will cross in front of the SDV when in reality it stops (false positives) are of greater importance than accurately characterizing the behavior of an actor 50 meters behind the SDV. We express the final reward as a simple linear combination of rewards

$$r(y,x)=\sum_{t}^{T}r_{\text{reach}}(y_{t},x)+r_{\text{route}}(y_{t},x)$$

Next, we explain both terms in detail.

So far we have presented our general framework, but we have not specified the particular perception and motion forecasting model we apply our prior knowledge loss to. We exploit a combination of the backbone feature extraction, object detection network, and graph propagation from SpAGNN [5], together with the mixture of Gaussians output parameterization of MTP [6]. This perception and prediction model takes a voxelized LiDAR point cloud and a raster map as input, extracts scene features using a backbone CNN, and applies Rotated Region of Interest Align (RRoI) [24] to extract per-actor features. After that, we define a fully-connected graph where the nodes correspond to traffic participants, and perform a series of graph-propagations to refine each actor’s representation by aggregating features from their neighbors, as proposed by [5]. Finally, an MLP header predicts the parameters of a multimodal distribution over future trajectories by using a mixture of Gaussians with full covariance for each actor:

$\displaystyle p(y|x)=\sum_{k}p(a^{k}|x)\prod_{t=1}^{T}p(y_{t}|\mu_{t}^{k}(x),\Sigma_{t}^{k}(x))$

Our method can be trained end-to-end using back-propagation and stochastic gradient descent. In particular, we minimize a multi-objective loss containing a classification and regression terms for object detection, a symmetric motion forecasting loss agnostic to the map or SDV, as well as the prior informed non-differentiable loss we described in Section III. The loss of each actor is a weighted sum of the multiple objectives.

$$\displaystyle\mathcal{L}=\alpha\cdot\mathcal{L}_{\text{det}}+\beta\cdot\mathcal{L}_{\text{symmetric}}+\gamma\cdot\mathcal{L}_{\text{prior}}$$

For the classification branch ($\mathcal{L}_{\text{cla}}$) of the detection header (background vs vehicle), we employ a binary cross entropy loss with hard negative mining. In particular, we select all positive examples from the ground-truth and 3 times as many negative examples from the rest of spatial locations. For box fitting ($\mathcal{L}_{\text{reg}}$), we apply a smooth L1 loss to each of the 5 parameters $(x,y,w,h,\phi)$ of the bounding boxes anchored to a positive example .

$$\displaystyle\mathcal{L}_{\text{det}}=\mathcal{L}_{\text{cla}}+\lambda\cdot\mathcal{L}_{\text{reg}}$$

Instead of directly optimizing the likelihood of the mixture model, we follow [6] in heuristically matching the closest mode to the ground-truth and only taking the negative log likelihood of that mode, while training the mode scores with a cross-entropy loss. This has been shown empirically [7] to be a more stable training objective than optimizing the mixture likelihood directly as in [25]. Thus we define

$\displaystyle\mathcal{L}_{\text{symmetric}}=-\sum_{k}\text{1}(k=\hat{k})[\log p(a^{k}|x)+\sum^{T}_{t}\log p(y^{k}_{t}|x)]$

where $\hat{k}=\operatorname*{arg\,min}_{k}\text{dist}(y^{k},y^{gt})$ is the mode whose mean is closest to the ground truth trajectory in euclidean distance.

Since the gaussian waypoints in this model are independent across time, we use a heuristic trajectory sampler at inference to draw smooth samples from this model (see Appendix -A for details). Note that during training we do not need this sampler since all the losses are formulated at the waypoint level. However, it is important to have temporally consistent samples for measuring the impact of our predictions in the downstream task of motion planning, since we approximate the heading into the future by finite differences between waypoints of the sample trajectories.

In this section, we first describe how we measure the motion forecasting ability of our method, namely how well it predicts the future behavior of all the traffic participants. We then introduce the comprehensive set of metrics that we use for evaluation in the downstream task of ego-motion planning. Despite being the final goal of the system, this task has generally been ignored in previous motion forecasting works. Next, we discuss the state-of-the-art baselines we compare against, report extensive quantitative results on two challenging, real-world datasets: ATG4D [2] and nuScenes [9], and showcase the differences of adding prior knowledge from a qualitative standpoint. Finally, we perform a thorough ablation study to justify our choices on how to incorporate prior knowledge. We defer our implementation details to Appendix -A.

In this paper we have proposed a novel framework to explicitly incorporate prior knowledge into probabilistic motion forecasts, while allowing to predict non-compliant behavior when there is evidence. Our method is general, and can be applied to any model that can generate trajectory samples and evaluate their marginal likelihood per actor. We have demonstrated the effectiveness of our approach in two challenging real-world datasets, significantly outperforming other state-of-the-art methods in both motion forecasting as well as in the downstream task of motion planning. Though we have chosen SpAGNN [5] as the base model here, our method is general. We plan on integrating our approach with more base models, with a particular interest in joint distributions over actors where we can also apply our prior knowledge about interactions, such as the fact that vehicles do not generally collide.