Towards High-Quality Temporal Action Detection with Sparse Proposals

We adopt the widely-used I3D network  [6] as our feature extractor for both RGB frames and optical flows, as it has the ability to model temporal information and proves its superior performance in action recognition. The last three stage outputs from I3D network are extracted to build the three-level hierarchical 3D feature maps. For a video clip with shape $(C,T,H,W)$, the temporal strides and spatial strides of the three-level features are [2, 4, 8] and [8, 16, 32], respectively. Finally, the same level features of RGB and flow streams are concatenated along the channel dimension.

After obtaining the three-level 3D feature maps from the backbone network, we first perform average spatial pooling to get the three-level 1D feature sequences. The features are further transformed by 1D convolutional layers with the output channel as 256. Then, the semantic information of high-level features is passed to low-level features in a top-down fashion. And a 1D max-pooling layer with stride as 2 is added upon the highest level feature. In this way, we construct a four-level feature pyramid with the minimal temporal resolution as $T/16$. Since the lower level features have a higher temporal resolution, they are responsible for predicting short action instances.

The action detection head receives the multi-level features generated by the temporal feature pyramid network (TFPN) and then predicts the temporal segments and action categories of action instances. The key elements of action detection head design are $N$ learnable proposal segments and corresponding proposal features. We set the number of proposal candidates as a small value (e.g., $N=50$), which should be larger than the maximum ground-truth action instances number for all videos clips in a dataset. The proposal segments are 2-d parameters, indicating the normalized center location and duration of a temporal segment. These proposal segments could be set as any size and placed randomly over the feature sequence during initialization, avoiding sophisticated candidate proposal design. The proposal feature encodes the rich instance information for each proposal candidate, playing a similar role as the object query in DETR  [5]. The main distinction lies in that every proposal feature only interacts with the corresponding Segment-of-Interest (SoI) feature instead of the global encoded feature, which could be seen as local attention.

In the action detection head, the proposal features first pass through the multi-head self-attention (MSA) module  [36] to model the relations between each other and then are fed into the sparse interaction module (SIM).

Figure 3 illustrates the sparse interaction between the $k$-th proposal feature and corresponding SoI feature in the sparse interaction module. Firstly, due to the ambiguous temporal boundary of an action instance, for the original proposal segment with the region $\left[t_{s}^{*},t_{e}^{*}\right]$, it will be expanded to $\left[t_{s}^{*}-t_{l}^{*}/\eta,t_{e}^{*}+t_{l}^{*}/\eta\right]$, where $t_{l}^{*}=t_{e}^{*}-t_{s}^{*}$ and $\eta$ is empirically as 5. Then, we use the expanded proposal segment to extract the SoI feature $f_{k}\in\mathbb{R}^{T^{{}^{\prime}}\times d}$ from temporal feature pyramid via 1D SoI Align operation  [43], where $T^{{}^{\prime}}$ is the aligned temporal resolution. For a proposal segment which represents a temporal length of $t$, the corresponding SoI feature is extracted from the $l$-th level feature determined by:

Meanwhile, the proposal feature $p_{k}\in\mathbb{R}^{d}$ produces two parameters with size $d\times d_{h}$ and $d_{h}\times d$ via linear transformation. Afterward, the extracted SoI feature performs matrix multiplication with these two parameters successively. Therefore, the interaction process can be seen as the SoI feature propagating through two 1D convolutional layers. This design helps our model can fully exploit the high-resolution features from the backbone intermediate layers due to its efficiency.

The $N$ outputs features from SIM, called segment features, are further fed into the two-layer feed-forward network (FFN) to obtain the final segment representation. Lastly, two parallel branches are built upon the action detection head to get the final classification scores and boundary regression predictions of $N$ action instances. The classification branch is a linear layer with Sigmoid activation for predicting the probability of each action class. And the regression branch consists of a 3-layer feed-forward network with ReLU activation for temporal boundary regression.

An important strategy used in the action detection head is the iterative refinement, which is a commonly-used technique  [4, 5, 48, 32] to improve the performance. We stack the action detection heads and get the predictions of each head. The predicted proposal segments and segment features at each stage serve as the initial proposal segments and proposal features for the next stage so as to be consistently refined. The auxiliary loss is added at intermediate stages to stabilize the training process. We show here the iterative refinement is also the key element towards high-quality temporal action detection.

Action recognition  [38, 6] has been comprehensively studied for a long period and I3D network achieves fairly high accuracy in predicting the action labels of a video clip. Thus, we add the action classification head to predict the video/clip-level action classes probabilities so as to further adjust the classification scores of predicted action instances. Specifically, we directly perform the average spatial-temporal pooling on the three-level 3D features output by the backbone network and feed the pooled feature to a linear layer with Sigmoid activation. The training targets are all action categories happening in the video clip.

The predicted action instances set $\mathbf{\Psi}_{p}$ generated by the action detection head contains $N$ samples, which is larger than the maximum number of ground-truth action instances in a dataset. As with DETR  [5], we expand the target set $\mathbf{\Psi}_{g}$ to the size $N$ by padding $\varnothing$ which indicates no action instance. Then, we adopt the set prediction loss on these two fixed-size sets. And we use the binary cross entropy loss for the action classification head. The overall training loss to optimize the whole network is constituted by these two parts:

The first part of Equation (2) is the set prediction loss. It first finds the optimal bipartite matching between predictions and ground-truth action instances by optimizing the matching cost. This provides a one-to-one label assignment pattern, i.e., only those predictions matched with the ground-truth action instances are denoted as positive samples. Therefore, for the $n$-th prediction $\psi_{n}$, there exists a unique instance $\hat{\psi}_{\sigma(n)}$ in target set to align with it. Finally, the set prediction loss is computed by the weighted sum of classification loss and regression loss. $\mathcal{L}_{\mathit{cls}}$ is the focal loss  [23] for action classification of action instances. $\mathcal{L}_{\mathit{L1}}$ and $\mathcal{L}_{\mathit{giou}}$ are L1 loss and generalized tIoU loss  [30] between the normalized segment locations of predicted samples and ground-truth instances. They together contribute to the boundary regression.

The inference process of the SP-TAD is simple. Given a video, $N$ predicted action instances associated with the classification probabilities and temporal segments are retrieved from the last action detection head. The classification probabilities are further adjusted by multiplying the video/clip-level action classes scores produced by the action classification head.

Most of the previous works  [19, 43, 46, 33] adopt an offline feature extractor to obtain the video features in advance and then design the network to process the fixed features. We hypothesize that fixed video features would limit the capacity of the current TAD methods because of the misalignment between different datasets. The recent works  [24, 18] jointly train the backbone but do not explicitly explore the effect of the unified backbone in the network.

We gradually increase the learning rate for training backbone while keeping the learning rate for other parts unchanged as $1e^{-4}$. Specifically, when the learning rate of backbone equals 0, the backbone could be seen as an offline feature extractor. From Table 4, we observe the consistent improvement by increasing the learning rate, which clearly proves that the unified backbone in the network would benefit the TAD task.

The previous works neglect the features output by the intermediate layers of the backbone. However, we contend that these features preserve rich temporal information and play a key role in high-quality temporal action detection. Here we study different kinds of features as the input to the action detection head:

• •

  “Single Level” means only using the last layer output of the backbone.

• •

  “High Level TFP” represents that building the four-level temporal feature pyramid upon the highest level features from the backbone.

• •

  “Inter Level TFP” is the four-level feature pyramid used in our model, which uses the intermediate layer outputs of the backbone to construct the hierarchical feature maps.

From Table 5, we find that “Single Level” feature performs even better than the “High Level TFP”. This is because most of the aligned SoI features are extracted from the higher levels of temporal feature pyramid and they have lower temporal resolution than the “Single Level” feature. These high-level features would lose the details of short action instances and thus degenerate the performance instead. A straightforward but effective solution is using the intermediate outputs from the backbone to construct a temporal feature pyramid. It achieves significant performance gain, especially for high tIoU thresholds, as illustrated in the last row of Table 5.

We study the effect of the number of stacked action detection heads in Table 6. The first row of Table 6 clearly shows that the model has an extremely poor performance of 43.6% on mAP@Avg without the iterative refinement process, while the performance would be greatly improved by increasing head number to 2, bringing a gain of 6.0% on mAP@Avg. This proves the effectiveness and necessity of iterative refinement. It also indicates that a good initialization for the proposal segments would reduce the prediction difficulty for the action detection head. As head number gradually increases, the performance gain becomes smaller and it saturates at the stage of 4. The iterative refinement is another important element for high-quality temporal action detection.

The action recognition results have achieved fairly high thanks to the development of strong 3D backbones. Therefore, we add an action classification head to predict the probability distribution of different actions. It could be used to rectify the classification scores predicted by the action detection head during inference phase. As illustrated in Table 7, it improves the mAP@Avg from 52.1% to 53.5% on THUMOS14 testing set, proving this strategy is an effective way to correct the classification error by action detection head.

In real applications, a long untrimmed video would contain several action instances with large duration variance. Therefore, applying the sliding windows on the video is a useful technique for accurate temporal action detection. We vary the length of sliding window from 224 to 320 while keeping the overlap ratio as 0.5 during inference. The results are summarized in Table 8. A window with the length of 256 can cover 98% action instances in all the videos, thus it would be a suitable choice. When the length decreases to 224, the network can still achieve strong performance but would falsely detect the over-long action instances. On the other hand, increasing the length of sliding window means that the action segments would take less proportion in a clip, which makes it harder to detect the short action instances. It is reasonable to expect a slight performance drop by adopting a larger window.