Group Activity Prediction with Sequential Relational Anticipation Model

Given $t_{0}$ observed frames, we first extract features of all the observed $t_{0}$ frames, and then apply ROIAlign [15] to extract the feature vectors of multiple people based on their positions $\left\{B_{1},B_{2},\cdots,B_{t_{0}}\right\}$($t\in\{1,\cdots,t_{0}\}$). Action features and positions of the $i$-th individual on the $t$-th frame are represented as $\mathbf{x}_{t}(i)$ and $\mathbf{b}_{t}(i)$ respectively. Afterwards, upon the individual dynamics, we follow [48] to explicitly model the pair-wise position relations and action relations of multiple people in the observed frames as two relation graphs $G^{\text{a}}_{t}\in\mathbb{R}^{N\times N}$ and $G^{\text{p}}_{t}\in\mathbb{R}^{N\times N}$ , respectively. Both of the two graphs have $N$ nodes representing $N$ people in the $t$-th frame. Given the $i$-th and $j$-th individuals, the edge of the action similarity graph $G^{\text{a}}_{t}(i,j)$ is computed by the cosine similarity and normalized by Softmax function. The edge on the position relation graph $G^{\text{p}}_{t}(i,j)$ is computed by the normalized Euclidean distance (denoted by $d(\cdot,\cdot)$):

Once the graphs are built, we obtain the structured representations for the group activity in the observed frames. We will also perform anticipation on the two graphs representing the group activity in the unobserved frames, which will be discussed below.

The observation encoder $\mathcal{E}$ is proposed to summarize spatiotemporal information of the complex relational dynamics of multiple people in partial observations containing $t_{0}$ frames. $\mathcal{E}$ learns to map $G^{\text{a}}_{1:t_{0}}$, $G^{\text{p}}_{1:t_{0}}$, and $X_{1:t_{0}}$ to a latent variable $Z_{0}$, by the spatio-temporal graph convolution network ST-GCN [50]. Specifically, it first performs spatial graph convolution [20] on the two graphs $G_{t}^{\text{p}}$ and $G_{t}^{\text{a}}$ for the $t$-th frame

and then performing temporal convolution [28] on every three consecutive frames to learn the latent variable $Z_{0}$. Here, $\sigma$ is ReLU activation, $W_{\text{p}}$ and $W_{\text{a}}$ are learnable weights, and $X_{t}$ is the action features of $N$ people. Latent variable $Z_{0}$ will be integrated in the sequential decoder, and guides its unrolling stages.

Different from ARG [48], our model captures the temporal patterns of people’s relations, which is useful for group activity prediction.

The performance of state-of-the-art action prediction methods [24, 23] is still limited especially when few beginning frames are given. This is mainly because they use a direct mapping from partial observation to the corresponding full observation in one pass, which is not powerful enough to deal with large visual variations between partial and full observations. In this paper, we propose a sequential decoder that progressively anticipates the group representation that is expected to contain rich discriminative information as the fully observed activity using $K$ unrolling stages (see Fig. 3). This allows us to create a more powerful model for group activity prediction.

Besides, different from individual action prediction methods [24, 45], people’s relations formulated as graphs using Eq. (1), are discriminative information for group activity. Moreover, the group activity varies overtime. It is necessary to predict group representations by anticipating relations in the unobserved stage. As described in Section 3.1, people’s relations can be inferred from their action similarity and relative positions. For example, a partial observation of a volleyball activity is given, which contains run-up of ace spikers and waiting gestures of their opponents. Our model is supposed to predict it as “spiking” by the cue that the players are moving towards net with their actions. Therefore, we develop a sequential decoder as a mixture of two graph auto-encoders: an activity auto-encoder $\mathcal{E}_{\text{a}}\text{-}\mathcal{D}_{\text{a}}$ for predicting activity representations and a position auto-encoder $\mathcal{E}_{\text{p}}\text{-}\mathcal{D}_{\text{p}}$ for predicting positions of multiple people. The two auto-encoders are coupled by the shared latent variables $Z_{0}$ learned from partial observations.

Activity auto-encoder $\mathcal{E}_{\text{a}}\text{-}\mathcal{D}_{\text{a}}$. Using $K$ activity auto-encoders, the proposed sequential decoder progressively anticipates the activity representation by $K$ unrolling stages. Each activity auto-encoder at the $k$-th stage ($k\in\{1,2,\cdots,K\}$) is fed with the output $\hat{X}_{k}$ of the activity auto-encoder at the previous $(k-1)$-th stage. We use the spatiotemporal action features at the last observed frame $t_{0}$ as the input of the activity auto-encoder at stage $k=1$. We encode the input $\hat{X}_{k}$ of current unrolling stage to a latent variable $Z^{\text{a}}_{k}$ by

and then decodes the activity representation $\hat{X}_{k+1}$:

where $U_{\text{ep}}$, $U_{\text{ea}}$, $U_{\text{dp}}$, $U_{\text{da}}$ are learnable parameters. $\hat{X}_{k+1}$ is the anticipated group features at the $k$-th stage, and is served as the input for the activity auto-encoder at the $(k+1)$-th stage. The anticipation of ${\hat{X}}_{k+1}$ is conditioned on latent variables $Z_{k}^{a}$ and $Z_{0}$, in order to both keep track of the short-term information of the previous unrolling stage and use the long-term spatiotemporal information in the partial observations. $G^{\text{a}}_{k}$ and $G^{\text{p}}_{k}$ are computed by the generated activity features and positions at the $k$-th stage using similar functions as Eq. (1), respectively (replacing time step $t$ by the stage $k$).

The benefits of the progressive anticipation using $K$ unrolling stages lie in two aspects. First, the temporal dependency of activity evolution is naturally built between successive stages. This allows us to naturally anticipate structured group activity representations for prediction purpose. Second, the prediction granularity can be controlled with the number of unrolling stages $K$. The case when $K=1$ is equivalent to the existing one-pass solution used in [24, 23].

Position auto-encoder $\mathcal{E}_{\text{p}}\text{-}\mathcal{D}_{\text{p}}$. As described in Section 3.1, the interactions between two people also depend on their relative positions. Thus, it is necessary to explicitly anticipate the positions of these people in group activity prediction.

Similar to activity auto-encoder, the proposed sequential decoder also performs $K$ unrolling stages for position prediction for a group of people using $K$ position auto-encoders. Each position auto-encoder at stage $k$ is fed with the output of its previous auto-encoder at stage $k-1$, and outputs the positions of people. Experimental results in Section 4.4 show that the anticipated future positions of people help improve performance of group activity prediction.

The position auto-encoder first encodes the positions $\hat{B}_{k}$ of multiple people to a latent variable $Z^{\text{p}}_{k}$ at stage $k$ through graph convolution [20]:

and then decodes the positions $\hat{B}_{k+1}$ for the next stage by

where $V_{\text{ep}}$, $V_{\text{ea}}$, $V_{\text{dp}}$, $V_{\text{da}}$ are learnable parameters. $G^{\text{p}}_{k}$ and $G^{\text{a}}_{k}$ are the same graphs used in the activity auto-encoder. The anticipation of $\hat{B}_{k+1}$ is conditioned on latent variables $Z_{k}^{\text{p}}$ and $Z_{0}$, in order to both keep track of the short-term position information of the previous unrolling stage and use the long-term spatiotemporal information in the partial observations.

Position prediction is also benefited by sequential prediction via several unrolling stages, since the prediction granularity can be controlled. Similar to the activity auto-encoder, the position auto-encoder at stage $k=1$ also takes the positions $B_{t_{0}}$ of people on the last observed frame as input. The activity auto-encoder and the position auto-encoder share the same graphs $G^{\text{p}}_{k}$ and $G^{\text{a}}_{k}$ and are both conditioned on the latent variable $Z_{0}$ (see Fig. 3).

SRAM returns both group activity and position representations at each of $K$ unrolling stages. The $K$-th stage corresponds to the full observation status, which contains the most discriminative information of an activity. We disregard all the outputs given by the activity autoencoders from the $1$-st to $(K-1)$-th stages, and perform max-pooling on the output $\hat{X}_{K+1}$ given by the activity autoencoder at the $K$-th stage as the group activity representations. The resulting feature vector is used for group activity prediction. Similarly, we directly use the output $\hat{B}_{K+1}$ given by the $K$-th position autoencoder to perform position prediction.

Adversarial loss. Inspired by [13], we encourage SRAM to generate representations corresponding to ground-truth full observations. We use two discriminators for the features generated by the sequential decoder. Discriminator $D_{1}$ is an activity classifier implemented by a softmax layer. $\mathcal{L}_{\text{cls}}$ is computed on the output of $D_{1}$. Discriminator $D_{2}$ is an adversarial regularizer and tells the difference between the generated group features $\hat{X}_{1:K}$ and group features of full videos $F_{1:K}(X)$. Using the adversarial loss, SRAM is encouraged to generate features that are indistinguishable from the group features of the corresponding full videos:

Note that the generated group representation $\hat{X}_{1:K}$ is computed by SRAM $S$ from the partial observation $X_{1:t_{0}}$.

Sequential reconstruction loss is proposed to align the predicted activity representations to become close to the ground-truth activity representations at each unrolling stage. Since our method has $K$-stage sequential prediction, it is necessary to encourage the predicted group representations $\hat{X}_{1:K}$ on each of the $K$ unrolling stages to become close to the ground-truth features at that timestamp. This is different from adversarial loss that only align the generated features to be close to ground-truth at full observation stage.

We train a separate ST-GCN $F(\cdot)$ as a recognition model to obtain the group activity representations of full videos $X$ for training. The resulting frame-wise group representations $F(X)$ are used to encourage the activity features of the $i$-th person generated at the $k$-th unrolling stage to be similar to the ground-truth features using

Here, $F_{k}(X,i)$ is the features of the $i$-th person of the full video at the $k$-th stage. This loss function sequentially computes the difference between the predicted features $\mathbf{\hat{x}}_{k}(i)$ (the $i$-th row on $\hat{X}_{1:K}$) for the $i$-th person at unrolling stage $k$ and the ground-truth features $F_{k}(X,i)$, mimicking how a partial observation is progressively approaching its corresponding full observation.

Position regression loss. We use the tracklets of individuals provided by [18] as the ground-truth of individuals positions. During training, we use the mean square error between the predicted positions and ground-truth positions at the $K$ unrolling stages as loss function:

where the predicted position $\hat{\mathbf{b}_{k}}(i)$ is the $i$-th row of $\hat{B}_{k}$, i.e., the $i$-th person’s position predicted by the sequential decoder at the $k$-th stage.

Model learning. During training, the overall objective function is written as a sum of sequential reconstruction loss $\mathcal{L}_{\text{rec}}$, adversarial loss $\mathcal{L}_{\text{GAN}}$, classification loss $\mathcal{L}_{\text{cls}}$ implemented by softmax loss, and position regression loss $\mathcal{L}_{\text{reg}}$:

Sequential relational anticipation model $\mathcal{(E,D)}$ and two discriminators ($D_{1},D_{2}$) are alternatively trained until convergence.

Group activity modeling and anticipation. Our SRAM captures the interactions of multiple people in the observation encoder, and anticipates their future relations by a sequential decoder. This is different from existing action prediction methods [24, 32] that can only predict the action of an individual. We believe such a novel method will pave the way for future research in other structured visual prediction.

Structured sequential prediction. Compared with group activity recognition methods [48, 32, 18], our method performs sequential prediction of group activity, in form of future positions and activity representations. Our activity prediction is also facilitated by explicitly predicting people’s future positions.

Activity evolution over time. Our sequential decoder progressively predicts future representations through several unrolling stages, which boosts performance when only few frames are observed. It is guided by a sequential reconstruction loss, mimicking how a partial observation is sequentially approaching its full observation and an adversarial loss to make the generated full observation features to become indistinguishable from the real full observation features.

Volleyball Dataset [19] consists of $4830$ video clips distributed in $8$ group activities, such as left spiking and right setting. Each clip has $41$ frames. [19] provides the players’ tracklets and splits the dataset into training, validation and testing sets. Existing group activity recognition methods [48, 19, 18, 32] use the middle $10$ frames of each video. To generalize it to prediction task, we extend it to use the middle $20$ frames as full observations, in order to model sequential dynamics. Note that the middle $20$ frames contain complete group activity executions, because athletes generally move quickly to complete a group activity, such as direct spiking in a volleyball game.

Collective Activity Dataset (CAD) [7] contains $44$ videos with $5$ group activities, including crossing, queueing, walking, talking and waiting. The group activities in CAD are labeled as the majority of people’s individual actions. We use the existing tracklet information and training/testing splits following [48]. The number of the frames in videos ranges from $100$ to $2000$. Following [32, 48, 4], we divide each video into $10$-frame video clips. This expands training and testing data to $1746$ and $765$ clips, respectively. CAD mainly contains periodic activities such as walking, in which significant changes can be seen in $10$ frames.

Following [48], we extract a $1024$-dimensional feature vector for each individual with tracklets provided by [19], using Inception-v3 [40] as backbone and ROIAlign [15]. We use three steps for training: First, Inception-v3 pretrained on ImageNet is fine-tuned on single frames by jointly predicting individual actions and group activities. Then, we freeze the backbone and finetune the recognition model $F(\cdot$) given full videos in the training set. The recognition model contain two ST-GCN layers [50], both with $256$-d hidden units. After that, we train the proposed model. The observation encoder has two layers ST-GCN with both $256$-d hidden units. The activity auto-encoder’s encoder has one graph convolution layer that encodes the input into $256$-d latent feature space. The position auto-encoder has two layer graph convolution by encoding the $2$-d positions into $64$-d space and then $256$-d latent space. During training, SRAM plus classifier $D_{1}$ and discriminator $D_{2}$ are alternatively updated.

The experiments are conducted with $10$ different observation ratios ranging from $10\%$ to $100\%$ of full videos length. The number of unrolling stages $K$ is set to $5$. We use stochastic gradient descent for optimization. For Volleyball dataset, the three steps are trained for $30$ epochs, $10$ epochs and $20$ epochs with learning rate $0.001$, $0.001$, $0.0001$ respectively. For Collective Activity Dataset, the three steps are trained for $20$ epochs, $50$ epochs and $10$ epochs with learning rate $0.0001$, $0.0001$, $0.0005$, respectively.

We compare our method with the state-of-the-art prediction methods LRCN [42], DeepSCN [23], IBoW and DBoW [34], KD [45], AAPNet [24] and state-of-the-art group activity recognition methods, including HRN [18], HDTM [19] ARG [48], SSU [4]. Following these methods’ original setting, LRCN and HDTM adopt the AlexNet [25] as the backbone. HRN, IBow, DBow, DeepSCN and original AAPNet use VGG-19. Our method follows ARG and SSU to use Inception-V3 method. HRN, HDTM, ARG, SSU and our method adopt the tracklets of players provided by [19]. To make a fair comparison, we extend state-of-the-art action prediction method AAPNet (“e-AAPNet” for simplification) to make use of tracking information and use Inception-V3 as backbone. We train all of the comparison methods using the parameters described in their original papers.

We perform detailed ablation studies on the Volleyball dataset to evaluate the contributions of the sequential prediction strategy, as well as the loss functions.