Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing

As the two most common and fundamental sensory information, visual and auditory data have attracted a large amount of research attention and derived many different audio-visual understanding tasks, such as audio-visual feature representation learning [23, 24, 25, 26, 27, 28, 29], audio-visual action recognition [1, 2, 3], sound source localization [30, 31, 32, 33], audio-visual video captioning [34, 35, 36], audio-visual event localization [9, 10, 11, 12, 13, 14] and audio-visual sound separation [4, 5, 6, 7, 8]. Most audio-visual learning methods are designed based on the assumption that audio and visual modalities are synchronized and temporal correlated. Concretely, [27, 25, 28, 29, 32] attempt to learn audio-visual feature representation jointly by utilizing the temporal alignment information between audio and visual modalities as the self-supervised guidance. In [26], a novel pretext task is proposed to extract correspondent feature for correlated video frame and audio, where the audio-visual feature pairs belonging to the same temporal snippet are gathered and the features from unpaired video snippets are separated. In [24], the unsupervised multimodal clustering information is utilized as the supervision for cross-modality feature correlation learning. To enhance the object detection capability, the correlation information between audio and object motion is taken into consideration [5, 37, 38, 39]. The methods mentioned above have achieved some progress by utilizing the synchronization between different modalities, which is not always satisfied in realistic scenes.

WS-AVVP aims to parse an arbitrary untrimmed video into a group of event instances associated with semantic categories, temporal boundaries and modalities (i.e., audio, visual and audio-visual), where only the video-level labels are provided as the supervision information for training. Tian et al. [15] first propose the WS-AVVP setting and design a hybrid attention network in a multimodal multiple instance learning (MMIL) pipeline, where the intra- and cross-modality attention strategies are utilized to capture cross-modality contextual information. Afterwards, Lin et al. [16] further explore cross-video and cross-modality complementary information to facilitate WS-AVVP, where both the common and diverse event semantics across videos are exploited to identify audio, visual and audio-visual events. Thereafter, to refine the event labels individually for each modality, Wu et al. [40] propose a novel method by swapping audio and visual modalities with other unrelated videos. JoMoLD [17] utilizes the cross-modality loss pattern to help remove the noisy event labels for each modality. Although the methods mentioned above have enhanced the cross-modality feature learning and mitigated the issue of modality-specific noisy labels, none of them have noticed the imbalanced multimodal feature learning.

Compared with the uni-modality learning, multimodal learning can integrate more information from different aspects. However, there always exists a discrepancy between different modalities, which makes the extraction and fusion of information from different modalities become more challenging. Concretely, the information volume and complexity in different modalities are often variant, thus the training difficulty of the networks for variant modalities is also different, which will lead to uncoordinated optimization issue [41, 19, 20, 21]: the dominated modality [18] conveying salient information will achieve more optimization attention and better performance during the whole training process, which will suppress the optimization progress of the other one. To cope with the above issue, Wang et al. [21] propose a metric to measure the overfitting-to-generalization ratio (OGR) and design a novel training scheme to minimize the OGR via an optimal blend of multiple supervised signals. To enhance the feature embedding capability of the suboptimal modality, Du et al. propose to distill reliable knowledge from the well-optimized model to help strengthen the optimization of the other one. Similar to our work, OGM [22] proposes to measure the optimization discrepancy between different modalities by calculating the ratio of predicted scores for the correct categories in different modalities, which is then utilized to modulate the gradients of different modality-specific models. However, OGM only takes the prediction scores for the correct event categories into consideration, which is not sufficient for reliable imbalance assessment. Consequently, we further consider the global prediction distribution on all categories. Meanwhile, to mitigate the negative effect of multimodal confusing calculation on our proposed DGM mechanism, we design the MSDU, which can cooperate with DGM for more significant improvement.

In this paper, we follow the standard protocol of WS-AVVP  [15]: Formally, given a multimodal training video sequence $\left\{a^{t},v^{t}\right\}_{t=1}^{T}$ with $T$ snippets, only the coarse video-level label $Y\in\left\{0,1\right\}^{C}$ is available during the training process, where $a^{t}$ and $v^{t}$ indicate the $t$-th audio and visual snippets and $C$ denotes the number of event categories. In the testing phase, the trained model should predict the snippet-level semantic category label $\mathbf{y}_{t}=\left\{y_{t}^{a},y_{t}^{v},y_{t}^{av}\right\}$ for audio, visual and audio-visual modalities respectively, where $y_{t}^{av}=y_{t}^{a}\times y_{t}^{v}$. In other words, the audio-visual events happen only when the corresponding audio and visual events belonging to the same semantic category occurr at the same time. Notably, the video-level event category label (multi-hot vector) can be obtained by aggregating the snippet-level event category annotations along the temporal dimension in each video. Due to the lack of fine-grained snippet-level supervision information during the training process, current WS-AVVP models are all designed in the multimodal multiple instance learning (MMIL) pipeline: in the training process, the snippet-level predictions from different modalities in the same video are aggregated to form the video-level prediction, which is supervised by the annotated video-level label.

The pipeline of our proposed framework is illustrated in Figure 2: Compared with our proposed framework, the traditional WS-AVVP pipeline (i.e., our proposed framework without the component in the light green background box) contains three main components including audio and visual feature encoders $\psi_{a/v}\left(\cdot\right)$ for snippet-level video feature learning, cross-attention mechanism $cro\_att(\cdot)$ for exchanging multimodal information, as well as multimodal attention $Atn_{a/v}\left(\cdot\right)$ and classification $\varphi_{a/v}\left(\cdot\right)$ modules for snippet-level video action prediction.

During the training process, an arbitrary training video containing audio $a$ and visual $v$ modalities is input into $\psi_{a/v}\left(\cdot\right)$ to produce the modality-specific snippet-level video feature, which is then fed into the cross-attention function $cro\_att(\cdot)$ to extract multimodal cross-attented feature representation. Afterwards, $Atn_{a/v}\left(\cdot\right)$ and $\varphi_{a/v}\left(\cdot\right)$ are utilized to generate the modality-specific attention weights and classification predictions. Ultimately, the attention weights and classification predictions in different modalities are combined together to produce the video-level action classification prediction, which is supervised by the video-level labels provided by the annotators. To optimize the model, the optimization objective of the traditional pipeline can be formulated as follows:

where $N$ denotes the number of training samples in a mini-batch and $Y\left[c\right]$ indicates the $c$-th element of $Y$. $P_{n}$ is the classification prediction for the $n$-th video, which is obtained via sigmoid function. Concretely, video-level classification prediction $P_{n}$ can be formally calculated as follows:

where $\alpha^{w}$ and $\alpha^{b}$ denote the weight and bias of modality-specific classifiers, $\pi(\cdot)$ is the formalization of feature embedding function including the modality-specific feature encoder $\psi(\cdot)$ and cross-attention block $cro\_att(\cdot)$, and $\theta$ indicates the hyper-parameter of $\pi(\cdot)$. In the following content, we omit the bias matrix $\alpha^{b}$ for more brevity. During the optimization process of our model, the gradient descent strategy is utilized to update the parameters of our model. Formally, the optimization processes of different parameters in our model can be formulated as follows (The superscript and subscript $a/v$ are omitted for brevity):

where $\lambda$ is the learning rate of the optimizer. Obviously, the common item in the above two equations is (Refer to Supplementary Material for detailed deduction of Eq. (5)):

From the above formulations, we can intuitively draw the conclusion that if the semantic information contained in the audio modality is much more obvious than the visual one, the audio will achieve higher predicted confidence scores. Furthermore, the back-propagation gradient (Eq. (5)) is more contributed by $\alpha^{w}_{a}\cdot\pi_{a}(a_{n};\theta_{a})$, which will make the audio achieve more optimization attention. Consequently, the visual modality will have a relatively lower prediction confidence and limited optimization efforts will be paid to it during the training process. Ultimately, although the training process of the whole multimodal model has converged, the modality containing relatively weak semantic information could not be fully optimized.

To cope with this issue, we propose a simple but effective dynamic gradient modulation (DGM) strategy (i.e., the component shown in the light green background box of Figure 2) to balance the feature learning of audio and visual modalities. Specifically, inspired by the factor that fully optimized model will predict higher classification scores for those correct categories and the discrepancy between the predicted scores for correct and wrong categories will be large, we assess the relative optimization progress between visual and audio modalities as follows:

where $s_{n}^{v}$ and $s_{n}^{a}$ are visual and audio classification confidence scores of the $n$-th sample, and $\triangle\bar{s}_{n}$ denotes the corresponding discrepancy between the average prediction scores of the correct and wrong event categories. Similarly, $\omega_{a-v}$ is the reciprocal of $\omega_{v-a}$. If the $\omega_{v-a}$ is larger than $1$, the optimizer will pay more attention to the visual modality, which will result in suboptimal audio feature learning. As a result, the balance coefficients of different modalities can be designed as follows:

where $\tanh\left(\cdot\right)$ denotes the activation function and $\gamma$ is a hyper-parameter managing the modulation degree. $\mu^{a}$ can be obtained in a similar way, but we omit the corresponding description for clarity. Afterwards, we utilize the balance coefficients $\mu$ to modify the gradients of different sub-networks during the back-propagation process:

where $W$ indicates the parameter to be optimized (i.e., $\theta$ and $\alpha^{w}$ in our model).

Moreover, according to [42, 43, 22], the back-propagation gradients in each batch follow a Gaussian distribution and an appropriately large gradient covariance will lead to better generalization ability. However, the gradient covariance modified by our DGM mechanism becomes $\mu^{2}\cdot\sigma^{2}(\frac{\partial{L}}{\partial{W}})$, which is smaller than the original one $\sigma^{2}(\frac{\partial{L}}{\partial{W}})$ because $\mu\in(0,1]$. To make up for this deficiency, we add an extra term to the modified gradient covariance as follows:

where $E(\cdot)$ and $\sigma^{2}(\cdot)$ are the expectation and covariance.

Obviously, in our proposed DGM mechanism, the core idea is to utilize the discrepancy between the predictions of different modalities to assess the imbalanced feature learning between audio and visual modalities, which is further applied to modulate the gradient of each modality-specific feature encoder during the optimization process. However, almost all existing WS-AVVP models [15, 16, 40, 17] are designed in the following pipeline as shown in Figure 3(a): audio $a$ and visual $v$ data are first input into the feature encoders $\psi_{a/v}\left(\cdot\right)$ to produce the modality-specific feature. Thereafter, the cross-modality attention mechanism $cro\_att(\cdot)$ is utilized to exchange the related information in different modalities and produced the cross-attented features $f_{a}$ and $f_{v}$. Afterwards, $f_{a}$ and $f_{v}$ are separately fed into the classifier $\varphi(\cdot)$ and attention module $Atn(\cdot)$ shared by audio and visual modalities to produce the snippet-level classification probabilities (i.e., $P_{a}$ and $P_{v}$) and temporal attention weights (i.e., $A_{a}$ and $A_{v}$) for different modalities. Ultimately, $P_{a}$, $P_{v}$, $A_{a}$ and $A_{v}$ are aggregated together to generate the video-level event classification prediction $P$, which is supervised by the video-level event category label provided by the annotators. Formally, the above pipeline can be written as follows:

where $`*^{\prime}$ and $`+^{\prime}$ denote the broadcast multiplication and element-wise addition, and $Agg_{T}(\cdot)$ is the aggregation operation along the temporal dimension. In the traditional case, $P_{a/v}$ and $A_{a/v}$ are utilized to calculate $s^{a/v}$ and $\triangle\bar{s}^{a/v}$ for the assessment of imbalanced feature learning between different modalities.

By going in depth into the traditional pipeline mentioned above, we can find that due to the cross-attention operation between audio and visual modalities, both $f_{a}$ and $f_{v}$ encode the information from different modalities. Consequently, the information of different modalities are confusing, which makes it hard to purely measure the imbalanced feature learning between different modalities by directly utilizing the attention $A_{a/v}$ and classification predictions $P_{a/v}$ produced based on the confusing multimodal feature $f_{a/v}$.

According to the above analysis, we modify the traditional pipeline and propose a modality-separated decision unit (MSDU) for more pure assessment of imbalanced feature learning between different modalities. Concretely, MSDU is embeded between the modality-specific feature encoders $\psi_{a/v}\left(\cdot\right)$ and cross-attention block $cro\_att(\cdot)$. Our proposed pipeline is shown in Figure 3(b). Obviously, in our pipeline, apart from our proposed MSDU (i.e., the component shown in yellow background box), the rest is similar to the traditional WS-AVVP pipeline: audio $a$ and visual $v$ data is first input into the modality-specific feature encoders to produce the modality-specific video feature $e_{a/v}$, which are then fed into the cross-attention module for more robust feature extraction. Afterwards, the cross-attented multimodal feature $f_{a}$ and $f_{v}$ are further taken as the inputs of modality-specific attention blocks $Atn_{a/v}(\cdot)$ and classification modules $\varphi_{a/v}(\cdot)$, which produce snippet-level action attention weights $A_{a/v}$ and classification scores $P_{a/v}$. To achieve more pure measurement of imbalanced multimodal feature learning, in our proposed MSDU, another group of modality-specific attention blocks $Atn_{a/v}^{ms}(\cdot)$ and classification modules $\varphi_{a/v}^{ms}(\cdot)$ are designed to predict the snippet-level action attentions $A_{a/v}^{ms}$ and classification scores $P_{a/v}^{ms}$ based on pure audio and visual modality features $e_{a/v}$ respectively. Formally, the pipeline of our method can be written as follows:

In our pipeline, $P_{a/v}^{ms}$ and $A_{a/v}^{ms}$ are utilized to calculate $s^{a/v}$ and $\triangle\bar{s}^{a/v}$ in Eq. (6). $P_{a/v}$ and $A_{a/v}$ are utilized as the final predcitions of our proposed model. In this manner, $P_{a/v}^{ms}$ and $A_{a/v}^{ms}$ are both based on single-modality data, which will lead to purer measurement of multimodal feature learning.

To comprehensively investigate the effectiveness of our proposed DGM strategy, we also conduct experiments on CREMA-D [51] and AVE [9] datasets: CREMA-D is an audio-visual benchmark for speech emotion recognition. In this dataset, 7442 video clips from 91 actors are collected. This dataset is divided into the training and validation subsets, which contain 6698 and 744 videos separately. AVE is an audio-visual benchmark for audio-visual event localization learning. In this benchmark, there are 4143 10-second videos from 28 event categories. In our experiments, the split of this dataset follows [9].

To comprehensively validate the effectiveness of our model, we compare it with different state-of-the-art methods, including the weakly-supervised sound event detection model TALNet [46], weakly-supervised video action detection models CMCS [48] and STPN [47], modified audio-visual event localization algorithms AVE [9] and AVSDN [49], as well as the recent weakly-supervised audio-visual video parsing methods HAN [15], MA [40], CVCM [16], MM-Pyramid [50] and JoMoLD [17]. For fair comparison, all WS-AVVP models are trained by using the LLP training set with the same videos and features.

Detailed experimental results of our proposed model and other state-of-the-art methods on the LLP testing subset are reported in Table I. Intuitively, our proposed model performs favorably against all other compared methods and achieves the best performance on all audio-visual video parsing sub-tasks under both the segment-level and event-level evaluation metrics. Concretely, compared with the most recent JoMoLD model, our proposed model achieves 1.48 average performance gains for Audio, Visual, Audio-visual, Type@AV and Event@AV tasks under the segment-level evaluation metric. Meanwhile, the average performance improvement for different sub-tasks under the event-level metric is 1.44. The above results significantly demonstrate the effectiveness of our proposed method.

Additionally, to comprehensively investigate the effectiveness of our proposed method, more experiments are conducted on different audio-visual benchmarks including CREMA-D and AVE. The corresponding experimental results are summarized in Table II and Table III. Consistent performance improvement verifies the effectiveness and good generalization of our proposed method.

Compared with ‘JoMoLD+DGM’, MSDU can bring additional 1.14 and 0.94 average performance improvements for different sub-tasks under the Segment-level and Event-level metrics respectively. Obviously, our proposed DGM mechanism can bring more performance gains for ‘JoMoLD+MSDU’ than JoMoLD. We attribute the experimental phenomenon to the reason that the calculations of different modalities are confused in JoMoLD, which is not beneficial for pure assessment of imbalanced feature learning between different modalities. However, in our designed MSDU block, the calculation of different modalities is separated and pure, which can boost the effectiveness of DGM.

To sum up, we can draw the following conclusions from the above experimental results: (1) Our proposed DGM mechanism can improve the audio-visual video parsing performance by balancing the feature learning between different modalities; (2) The MSDU structure can effectively separate the calculation of different modalities, thus further promote the effectiveness of our proposed DGM mechanism. (3) Our proposed DGM mechanism can be adapted to different backbone structures, achieving consistent performance improvement in different WS-AVVP sub-tasks.