Proposal-Free Temporal Action Detection via Global Segmentation Mask Learning

Multi-class action classification Given the $t$-th snippet $E^{s}(t)\in\mathbb{R}^{c}$ (i.e, the $t$-th column of $E^{s}$), our classification branch predicts the probability $\bm{p}_{t}\in\mathbb{R}^{(K+1)\times 1}$ that it belongs to one of $K$ target action classes or background. This is realized by a 1-D convolution layer $H_{c}$ followed by a softmax normalization. Since a video has been encoded into $T^{s}$ temporal snippets, the output of the classification branch can be expressed in column-wise as:

Global segmentation mask inference In parallel to the classification branch, this branch aims to predict a global segmentation mask for each action instance of a video. Each global mask is action instance specific and class agnostic. For a training video, all temporal snippets of a single action instance are assigned with the same 1D global mask $\in\mathbb{R}^{T\times 1}$ for model optimization (refer to Fig. 3(a)). For each snippet $E^{s}(t)$, it outputs a mask prediction $\bm{m}_{t}=[q_{1},\cdots,q_{T}]\in\mathbb{R}^{T^{s}\times 1}$ with the $k$-th element $q_{k}\in[0,1]$ indicating the foreground probability of $k$-th snippet conditioned on $t$-th snippet. This process is implemented by a stack of three 1-D conv layers as:

where the $t$-th column of $\bm{M}$ is the segmentation mask prediction at the $t$-th snippet. With the proposed mask signal as learning supervision, our TAGS model can facilitate context-aware representation learning, which brings clear benefit on TAD accuracy (see Table 5).

Remarks: Actionness [18, 52] is a popular localization method which predicts a single mask in shape of $\in\mathbb{R}^{T\times 1}$. There are several key differences between actionness and TAGS: (1) Our per-snippet mask model TAGS focuses on a single action instance per snippet per mask so that all the foreground parts of a mask are intrinsically related; In contrast, actionness does not. (2) TAGS breaks the single multi-instance 1D actionness problem into a multiple 1D single-instance mask problem (refer to Fig. 3(a)). This takes a divide-and-conquer strategy. By explicitly segmenting foreground instances at different temporal positions, TAGS converts the regression based actionness problem into a position aware classification task. Each mask, associated with a specific time $t$, focuses on a single action instance. On the other hand, one action instance would be predicted by multiple successive masks. This predictive redundancy, simply removable by NMS, provides rich opportunities for accurate detection. (3) Whilst learning a 2D actionness map, BMN [18] relies on predicting 1D probability sequences which are highly noisy causing many false alarms. Further, its confidence evaluation cannot model the relations between candidates whilst our TAGS can (Eq. (7)). Lastly, our experiments in Table 8 validate the superiority of TAGS over actionness learning.

Ground-truth labels. To train TAGS, the ground-truth needs to be arranged into the designed format. Concretely, given a training video with temporal intervals and class labels (Fig. 3(a)), we label all the snippets (orange or blue squares) of a single action instance with the same action class. All the snippets off from action intervals are labeled as background. For an action snippet of a particular instance, its global mask is defined as the video-length binary mask of that action instance. Each mask is action instance specific. All snippets of a specific action instance share the same mask. For instance, all orange snippets (Fig. 3(a)) are assigned with a $T$-length mask (eg. $m_{24}$ to $m_{38}$) with one in the interval of $[q24,q38]$.

Learning objectives. The classification branch is trained by a combination of a cross-entropy based focal loss and a class-balanced logistic regression loss [12]. For a training snippet, we denote $y$ the ground-truth class label, $\bm{p}$ the classification output, and $\bm{r}$ the per-class regression output obtained by applying sigmoid on top of $H_{c}$ in Eq. (3) (discarded at inference time). The loss of the classification branch is then written as:

where $\gamma=2$ is a focal degree parameter, $\alpha=10$ is a class-balancing weight, and $\mathcal{N}$ specifies a set of hard negative classes at size of $K/10$ where $K$ is the toTAD action class number. We set the loss trade-off parameter $\lambda_{1}=0.4$.

Mask predictive redundancy Although the mask loss Eq. (6) above treats the global mask as a 2D binary mask prediction problem, it cannot always regulate the behaviour of individual 1D mask within an action instance. Specifically, for a predicted mask $m_{t}$ at time $t$, thresholding it at a specific threshold $\theta_{j}\in{\Theta}$ can result in binarized segments of foreground and background: $\pi[j]=\{(q_{s}^{i},q_{e}^{i},z_{i})\}_{i=1}^{L}$ where $q^{i}_{s}$ and $q^{i}_{e}$ denotes the start and end of $i$-th segment, and $z_{i}\in\{0,1\}$ indicates background or foreground. For a mask corresponding to an action snippet, ideally at least one of these $\{\pi[j]\}$ should be closer to the ground truth. To explore this redundancy, we define a prediction scoring criterion with the outer-inner-contrast [32, 17] as follows:

$l_{i}=q^{i}_{e}-q^{i}_{s}+1$ is the temporal length of $i$-th segment, $\delta$ is a weight hyper-parameter which is set to 0.25. We obtain the best prediction with the maximal score as $j^{*}=argmax(\mathbb{R}(\pi[j]))$ (see Fig. 4(c)). Higher $\mathbb{R}(\pi[j^{*}])$ means a better prediction. To encourage this best prediction, we design a prediction promotion loss function:

where we set $\beta=2$ for penalizing lower-quality prediction stronger. We average this loss across all snippets of each action instance per training video.

Classification-mask consistency In TAGS, there is structural consistency in terms of foreground between class and mask labels by design (Fig. 4(a)). To leverage this consistency, we formulate a feature consistency loss as:

where $\hat{F}_{clf}=topk(argmax((P_{bin}*E_{p})[:K,:]))$ is the features obtained from the top scoring foreground snippets obtained from the thresholded classification output $P_{bin}:=\eta(P-\theta_{c})$ with $\theta_{c}$ the threshold and $E_{p}$ obtained by passing the embedding $E$ into a 1D conv layer for matching the dimension of $P$. The top scoring features from the mask output $M$ are obtained similarly as: $\hat{F}_{mask}=topk(\sigma(1DPool(E_{m}*M_{bin})))$ where $M_{bin}:=\eta(M-\theta_{m})$ is a binarization of mask-prediction $M$, $E_{m}$ is obtained by passing the embedding $E$ into a 1D conv layer for matching the dimension of $M$, $*$ is element-wise multiplication, $\eta(.)$ is the binarization function, and $\sigma$ is sigmoid activation. Our intuition is that the foreground features should be closer and consistent after the classification and masking process (refer to Fig. 4(d)).

Overall objective The overall objective loss function for training TAGS is defined as: $L=L_{c}+L_{m}+L_{pp}+L_{fc}$. This loss is calculated for each temporal scale $s$ and finally aggregated over all the scales.

Results on ActivityNet From Table 1, we can make the following observations: (1) TAGS with I3D feature achieves the best result in average mAP. Despite the fact that our model is much simpler in architecture design compared to the existing alternatives. This validates our assumption that with proper global context modeling, explicit proposal generation is not only redundant but also less effective. (2) When using the relatively weaker two-stream (TS) features, our model remains competitive and even surpasses I3D based BU-TAL [51], A2Net [47] and the very recent ContextLoc [56] and MUSES [21] by a significant margin. TAGS also surpasses a proposal refinement and strong G-TAD based approach CSA [33] on avg mAP. (3) Compared to RTD-Net which employs an architecture similar to object detection Transformers, our TAGS is significantly superior. This validates our model formulation in exploiting the Transformer for TAD.

Results on THUMOS14 Similar conclusions can be drawn in general on THUMOS from Table 1. When using TS features, TAGS achieves again the best results, beating strong competitors like TCANet [39], CSA [33] by a clear margin. There are some noticeable differences: (1) We find that I3D is now much more effective than two-stream (TS), e.g., 8.8% gain in average mAP over TS with TAGS, compared with 0.6% on ActivityNet. This is mostly likely caused by the distinctive characteristics of the two datasets in terms of action instance duration and video length. (2) Our method achieves the second best result with marginal edge behind MUSES [21]. This is partly due to that MUSES benefits from additionally tackling the scene-changes. (3) Our model achieves the best results in stricter IOU metrics (e.g., IOU@0.5/0.6/0.7) consistently using both TS and I3D features, verifying the effectiveness of solving mask redundancy.

Computational cost comparison One of the key motivations to design a proposal-free TAD model is to reduce the model training and inference cost. For comparative evaluation, we evaluate TAGS against two representative and recent TAD methods (BMN [18] and G-TAD [46]) using their released codes. All the methods are tested on the same machine with one Nvidia 2080 Ti GPU. We measure the convergence time in training and average inference time per video in testing. The two-stream video features are used. It can be seen in Table 3 that our TAGS is drastically faster, e.g., $20/25\times$ for training and clearly quicker – $1.6/1.8\times$ for testing in comparison to G-TAD/BMN, respectively. We also notice that our TAGS needs less epochs to converge. Table 3 also shows that our TAGS has the smallest FLOPs and the least parameter number.

Transformers vs. CNNs We compare our multi-scale Transformer with CNN for snippet embedding. We consider two CNN designs: (1) a 1D CNN with 3 dilation rates (1, 3, 5) each with 2 layers, and (2) a multi-scale MS-TCN [14], and (3) a standard single-scale Transformer [36]. Table 5 shows that the Transformers are clearly superior to both 1D-CNN and a relatively stronger MS-TCN. This suggests that our global segmentation mask learning is more compatible with self-attention models due to stronger contextual learning capability. Besides, multi-scale learning with Transformer gives $0.4\%$ gain in avg mAP validating the importance of larger snippets. As shown in Table 5, the gain almost saturates from 200 snippets, and finer scale only increases the computational cost.

Proposal-based vs. proposal-free We compare our proposal-free TAGS with conventional proposal-based TAD methods BMN [2] (anchor-free) and R-C3D [42] (anchor-based) via false positive analysis [1]. We sort the predictions by the scores and take the top-scoring predictions per video. Two major errors of TAD are considered: (1) Localization error, which is defined as when a proposal/mask is predicted as foreground, has a minimum tIoU of 0.1 but does not meet the tIoU threshold. (2) Background error, which happens when a proposal/mask is predicted as foreground but its tIoU with ground truth instance is smaller than 0.1. In this test, we use ActivityNet. We observe in Fig. 5 that TAGS has the most true positive samples at every amount of predictions. The proportion of localization error with TAGS is also notably smaller, which is the most critical metric for improving average mAP [1]. This explains the gain of TAGS over BMN and R-C3D.

Direction of improvement analysis Two subtasks are involved in TAD – temporal localization and action classification, each of which would affect the final performance. Given the two-branch design in TAGS, the performance effect of one subtask can be individually examined by simply assigning ground-truth to the other subtask’s output at test time. From Table 7, the following observations can be made: (1) There is still a big scope for improvement on both subtasks. (2) Regarding the benefit from the improvement from the other subtask, the classification subtask seems to have the most to gain at mAP@0.5, whilst the localization task can benefit more on the average mAP metric. Overall, this analysis suggests that further improving the efficacy on the classification subtask would be more influential to the final model performance.

Analysis of components We can see in Table 7 that without the proposed segmentation mask branch, the model will degrade significantly, e.g., a drop of 7.6% in average mAP. This is due to its fundamental capability of modeling the global temporal structure of action instances and hence yielding better action temporal intervals. Further, for TAGS we use a pre-trained UntrimmedNet (UNet) [37] as an external classifier instead of using the classification branch, resulted in a 2-stage method. This causes a performance drop of 4.7%, suggesting that both classification and mask branches are critical for model accuracy and efficiency.

Global mask design We compare our global mask with previous 1D actionness mask [18, 43]. We integrate actionness with TAGS by reformulating the mask branch to output 1D actionness. From the results in Table 8, we observe a significant performance drop of $11.5\%$ in mAP@0.5 IOU. One reason is that the number of action candidates generated by actionness is drastically limited, leading to poor recall. Additionally, we visualize the cosine similarity scores of all snippet feature pairs on a random ActivityNet val video. As shown in Fig. 8, our single-instance mask (global mask) design learns more discriminating feature representation with larger separation between background and action, as compared to multi-instance actionness design. This validates the efficacy of our design in terms of jointly learning multiple per-snippet masks each with focus on a single action instance.