Non-Probability Sampling Network for Stochastic Human Trajectory Prediction

Convolutional neural network (CNN)-based approaches using Gaussian distribution have improved the efficiency of pedestrian trajectory prediction. Social-LSTM [1], a pioneering model in this field, predicts a bivariate Gaussian distribution consisting of five parameters for the observed trajectories of pedestrians. However, it has a limitation when inferring single paths, since it only selects the best one sample from the distribution in inference time. Follow-up works [55, 39, 49, 50] predict multiple paths by sampling multiple next coordinates based on predicted distributions.

As another methodology, a generative model is introduced to predict realistic future paths. Social-GAN [13] firstly uses a generative framework that recursively infers future trajectory. The benefit of GAN is that it generates various outputs according to latent vectors. As a result, inter-personal, socially acceptable and multimodal human behaviors are accounted for in the pedestrian trajectory prediction. Such a research stream encourages to define a variety loss which calculates for the best prediction among multiple samples for diverse sample generation. [22, 46, 51, 6, 10, 17].

Similarly, there have been attempts to predict diverse future generations using CVAE frameworks. DESIRE [23] uses a latent variable to account for the ambiguity of future paths and learns a sampling model to produce multiple hypotheses of future trajectories from given observations. This approach provides a diverse set of plausible predictions without the variety loss, and shares inspiration to objectives in many CVAE-based models [47, 34, 60, 32, 18].

All of these methods include a random sampling process and are sensitive to bias, due to the fixed number of samples, as above mentioned. In addition, current state-of-the-art models with CVAE frameworks outperform Gaussian distribution-based methods [39, 49]. In this study, we analyze these phenomena with respect to the bias of stochastic trajectory prediction, and show that the Gaussian distribution-based approaches achieve noticeable performance improvements by minimizing the bias, even better than the CVAE-based methods. Lastly, we mention a recent deterministic approach [63] that predicts multiple trajectories, which is beyond the scope of this paper.

Some works account for the transformation of latent spaces by using prior trajectory information. PECNet [34] for example uses a truncation trick in latent space to adjust the trade-off between the fidelity and the variety of samples. In their learning approach, both IDL [28] and Trajectron++ [47] predict the mean and standard deviation of a latent distribution in an inference step. Rather than directly predicting the distribution parameters, AgentFormer [61] uses a linear transform of Gaussian noise to produce the latent vector. These methodologies still run the risk of bias because of the random sampling of the latent vectors. In the present work, we aim to reduce the bias using a discrepancy loss of a set of sampled latent vectors.

Pioneering works have introduced the concepts of social-pooling [1, 13, 51] and social-attention mechanisms [55, 62, 27] to capture the social interactions among pedestrians in scenes. Recently, Graph Neural Network (GNN)-based approaches [17, 22, 39, 30, 49, 27, 2] have been introduced to model agent-agent interactions with graph-based policies. In the GNN-based works, pedestrians are regarded as nodes of the graph, and their social relations are represented as edge weights. Social-STGCNN [39] presents a Graph Convolutional Network (GCN) [21]-based trajectory prediction which aggregates the spatial information of distances among pedestrians. Graph Attention Networks (GATs) [54] implicitly assign more weighting to edges with high social affinity on the pedestrian graph [17, 22, 52, 49, 60]. Multiverse [30] and SimAug [29] utilize GATs on 2D grids to infer feasible trajectories. Unlike these previous works, where GATs are used in the encoding process, we apply a GAT framework to a sampling process on the latent space to make a decoder predict future paths more accurately.

(Quasi-) Monte Carlo is a computational technique for numerical experiment using random numbers. Exploiting the random numbers allows one to approximate integrals, but this is highly error prone. The error directly depends on the random sampling methods from probability distributions. QMC sampling is developed with quasi-random sequences, known as low-discrepancy sequences [40] and is generated in a deterministic manner. It is widely utilized for many computer vision tasks, such as depth completion [56], 3D reconstruction [57, 11], motion tracking [65] and neural architecture search [8, 9]. We firstly apply QMC sampling to ensure uniform coverage of the sampling spaces for pedestrian trajectory prediction. Note that the sequence is uniformly distributed if the discrepancy tends to be zero, as the number of samples goes to infinity.

We formulate the pedestrian trajectory prediction task as a multi-agent future trajectory generation problem conditioned on their past trajectories. To be specific, during the observation time frames $1\kern-2.58334pt\leq\kern-2.58334ptt\kern-2.58334pt\leq\kern-2.58334ptT_{obs}$, there are $L$ pedestrians in a scene. The observed trajectory sequence is represented as $\smash{X_{l}^{1:T_{obs}}}\!\!=\!\{X_{l}^{t}|t\in[1,...,T_{obs}]\}$ for $\forall l\!\in\![1,...,L]$, where $x_{l}^{t}$ is the spatial coordinate of each pedestrian $l$ at time frame $t$. With the given observed sequence, the goal of the trajectory prediction is to learn potential distributions to generate $N$ plausible future sequences $\smash{\hat{Y}_{l}^{1:T_{pred}}}\!\!=\!\{\hat{Y}_{l,n}^{t}|t\in[1,...,T_{pred}],n\in[1,...,N]\}$ for all $L$ pedestrians.

The generated trajectory $\smash{\hat{Y}_{l}^{1:T_{pred}}}$ comes from a distribution of possible trajectories which are constructed by pedestrians’ movements based on social forces (Fig. 3). $\mathcal{T}_{truth}$ is an expectation value computed with a plausible trajectory distribution, and $\mathcal{T}_{gen}$ is calculated with $\smash{\hat{Y}_{l}^{1:T_{pred}}}$ of $N$ which are independent and identically distributed (IID) random samples, i.e. the term is random if one uses different samples to generate trajectories. The expectation $\mathcal{T}_{gen}$ is a Monte Carlo estimate of integral, i.e. relevant expectation.

Suppose that the expectation we want to compute from the trajectory distribution is $\smash{I(\tau)=\int_{[0,1]^{s}}\tau(x)q(x)dx}$ which is the expected value of $\tau(x)$ for random variable $x$ with a density $q$ on $s$-dimensional unit cube $[0,1]^{s}$. Then, the Monte Carlo estimator for the generated trajectory distribution with $N$ samples can be formulated as below:

$$\hat{I}(\tau)=\hat{I}_{N,s}(\tau)=\frac{1}{N}\sum_{i=1}^{N}\tau(x_{i}),\vspace{-2mm}$$

(1)

$$Pr(\lim_{n\to\infty}\hat{I}(\tau)=I(\tau))=1,\vspace{-0.5mm}$$

(2)

where $Pr(\cdot)$ denotes a probability.

By the Strong Law of large numbers [12], the MC estimate converges to $I(\tau)$ as the number of samples $N$ increases without bound. Now, we assume that $\tau(x)$ has a finite variance $K(\tau)$ and define the error $\alpha$ as below:

$$\alpha=\hat{I}_{N,s}(\tau)-I(\tau),\vspace{-1mm}$$

(3)

$$\mathbb{E}[\alpha]=0,\quad var(\alpha)=\frac{K(\tau)}{N},\vspace{-0.5mm}$$

(4)

where $\mathbb{E}$ is an expectation and $K(t)$ is $\int(\tau(x)-I(\tau))^{2}q(x)dx$. Note that the $K(\tau)$ is non-negative and depends on the function being integrated. The algorithmic goal is to specify the procedure that results in lower variance estimates of the integral.

Now consider a function of the generator $F$, which is sufficiently smooth, in a Monte Carlo integral $I(\tau)$. We apply the Taylor series expansion of $F(I(\tau)\!+\!\alpha)$ as follows:

$$F(\hat{I}_{N,s}(\tau))=F(I(\tau)\!+\!\alpha)\qquad\qquad\qquad\qquad\qquad\quad\vspace{-2.5mm}$$

(5)

$$\qquad\qquad\quad\;\approx F(I(\tau))\!+\!\alpha F^{\prime}(I(\tau))\!+\!\alpha^{2}\frac{F^{\prime\prime}(I(\tau))}{2}\!+\!O(\alpha^{3}).\vspace{-2mm}$$

The QMC method utilizes a low discrepancy sequence including the Halton sequence [14] and the Sobol sequence [19]. Inspired by [42], we select a Sobol sequence which not only shows consistently better performances than the Halton sequence, but also is up to 5 times faster than the MC method, even with lower error rates.

From the view of numerical analysis, an inequality in [41] proves that low-discrepancy sequences guarantees more advanced sampling in Eq. 2 with fewer integration errors as below:

$$|\alpha|\leqq V(\tau)~{}D^{*}_{N},$$

(7)

where $V(\tau)$ is a total variation of function $\tau$ which is bounded variation, and $D^{*}_{N}$ is the discrepancy of a sequence for the number of samples $N$. The inequality shows that a deterministic low-discrepancy sequence can be much better than the random one, for a function with finite variation. In the mathematics community, it has been proven that the Sobol sequences have a rate of convergence close to $O((logN)^{s}/N)$; for a random sequence it is $O(\sqrt{log(logN)/N})$ in [42, 41]. For faster convergence, $s$ needs to be small and $N$ large (e.g., $N\!>\!2^{s}$). As a result, the low discrepancy sequences have lower errors for the same number of points ($N\!=\!20$) as shown in Tab. 1.

As an example, since $x_{i}$ are IID samples from a uniformly distributed unit box for MC estimates, the samples tend to be irregularly spaced. For QMC, as $x_{i}$ comes from a deterministic quasi-random sequence whose point samples are independent, they can be uniformly spaced. This guarantees a suitable distribution for pedestrian trajectory prediction by successively constructing finer uniform partitions. Fig. 4 displays a plot of a moderate number of pseudo-random points in 2-dimensional space. We observe regions of empty space where there are no points generated from the uniform distribution, which produce results skewed towards the specific destinations. However, the Sobol sequence yields evenly distributed points to enforce prediction results close to socially-acceptable paths.

Unfortunately, low-discrepancy sequences such as the Sobol sequence are deterministically generated and make the trajectory prediction intractable when representing an uncertainty of pedestrians’ movements with various social interactions. Adding randomness into the Sobol sequence by scrambling the sequence’s base digits [3] is a solution to this problem. The resultant sequence retains the advantage of QMC method, even with the same expected value. Accordingly, we utilize the scrambled Sobol sequence to generate pedestrian trajectories to account for the feasibility, the diversity, and the randomness of human behaviors.

In contrast to stochastic sampling, purposive sampling, one of the most common non-probability sampling techniques [4], relies on the subjective judgment of an expert to select the most productive samples rather than random selection. This approach is advantageous when studying complicated phenomena in in-depth qualitative research [37].

Since most people walk to their destinations using the shortest path, a large portion of labeled attributes in public datasets [43, 25] are straight paths. Generative attribute models learn the probabilistic distributions of social affinity features for the attribute of straight paths. However, due to the multimodal nature of human paths, the models must generate as many diverse and feasible paths as possible, using only a fixed number of samples. As a possible solution, we can purposively include a variety of samples on turning left/right and detouring around obstacles. In purposive sampling, a maximum variation is beneficial for multimodal trajectory prediction, when examining the diverse ranges of pedestrians’ movements. We make this process a learnable method, aiming to generate $N$ heterogeneous trajectory samples with prior knowledge of past trajectories.

We propose NPSN which substitutes the random sampling process of existing models with a learnable method. NPSN works as purposive sampling, which relies on the past trajectories of pedestrians when selecting samples in the distribution. As a result, when predicting a feasible future trajectory, a past trajectory can be used for the sampling process while also embedding informative features as a guidance. Unlike existing works [34, 28, 47, 61] that impose a restriction in the sampling space by limiting a distribution, we design all of the processes in a learnable manner.

However, we observe that all $N$ sample points are sometimes closely located near its ground-truth as learning progresses. This is a common problem in purposive sampling, because certain samples can be over-biased due to data imbalance, i.e. a large portion of the trajectory moving along one direction of the walkway. For this reason, we introduce a novel discrepancy loss to keep the $N$ sample points with low-discrepancy, as below:

$$\mathcal{L}_{disc}=\frac{1}{LN}\sum_{l=1}^{L}\sum_{i=1}^{N}-\log\min_{\begin{subarray}{c}j\in[1,...,N]\\ j\neq i\end{subarray}}||S_{l,i}-S_{l,j}||.\vspace{-2mm}$$

(9)

The objective of discrepancy loss is to maximize distances among the closest neighbors of $N$ samples. If the distance is closer, the loss imposes a higher penalty to ensure their uniform coverage of the sampling space.

The final loss function is a linear combination of both the distance and the discrepancy loss $\mathcal{L}=\mathcal{L}_{dist}+\lambda\mathcal{L}_{disc}$. We set $\lambda=1e\!-\!2$ to balance the scale of both terms.

We also observe that the Gaussian-based models show a large performance gain over the GAN- and CVAE-based models. There are two reasons for the performance gains induced by the QMC method: 1) The dimension of the sampling space ($s\!=\!2$) in the Gaussian-based models is relatively smaller than other models (i.e. $s\!=\!8$, $16$ or $25$). According to [24], for large dimensions $s$ and a small number of samples $N$, the sampling results from a low-discrepancy generator may not be good enough over randomly generated samples. The Gaussian-based model thus yields promising results compared to one which has larger sampling dimensions. 2) The performance improvements depend on the number of layers in networks (shallower is better): The CVAE and GAN-based models are composed of multiple layers. By contrast, the Gaussian-based models have only one layer which acts as a linear transformation between the predicted trajectory coordinates and final coordinates. To be specific, in the transformation, sampled independent 2D points are multiplied with the Cholesky decomposed covariance matrix and shifted by the mean matrix. Here, the shallow layer of the Gaussian-based models directly reflects the goodness of the QMC sampling method, rather than deeper layers which can barely be influenced by the random latent vector in the inference step.

In the CVAE based approaches, PECNet [34] shows a larger performance improvement (up to 41%) than that of Trajectron++ [47]. Since PECNet directly predicts a set of destinations through the latent vector, NPSN is compatible with its inference step. On the other hand, NPSN seems to produce less benefit with the inference step of Trajectron++ because it predicts the next step recurrently and its sample dimension is relatively large ($s=25$).

The generative models with variety loss, Social-GAN and STGAT, show relatively small performance improvements, compared to the others. For some datasets, the FDE values of STGAT are lower than those of MC and QMC when using our NPSN. This seems to suggest that NPSN fails to learn samples close to ground-truth trajectories due to the common entanglement problem of latent space [7, 20].

As we described in the Fig. 4, the QMC method generates a more realistic trajectory distribution than the MC method. However, due to the limitations of the dataset, the generated trajectories of the baseline network are biased toward a straight path. On the other hand, NPSN sampling method alleviates the problem by selecting the point near the ground-truth in the latent space. As a result, the human trajectory model with NPSN not only generates well-distributed samples with finite sampling pathways, but also represents the feasible range of human’s movements.

In addition, we report a benchmark result on ETH/UCY dataset in Tab. 2. It is noticeable that all the baseline models exhibit better performances with our NPSN. In particular, when NPSN is incorporated into the combinational approach of Trajectron++ [47] and NCE [32], it achieves the best performances on the benchmark. Our NPSN is trained to only control the latent vector samples for the baseline models, and synergizes well with the inference step that comes after both the initial prediction of Trajectron++ and the collision avoidance of NCE.