Audio-based Near-Duplicate Video Retrieval with Audio Similarity Learning

To generate audio-based descriptors, we first extract the Mel-filtered spectrogram from the audio of videos. All audio signals are resampled at $44.1$ kHz sampling frequency. For the spectrogram generation, we use 128 Mel-bands and a window size of $23$ ms (1024 samples at $44.1$ kHz) with an overlap of $11.5$ ms (512 hop size). The generated spectrograms are divided into overlapping time segments of $2$ seconds with $t$ seconds time step. We consider $t$ as a system hyperparameter and we measure its effect on Section V-A.

Further, we feed the generated spectrogram segments to a feature extraction CNN designed for transfer learning, proposed by Kumar et al. [2]. The CNN model is trained on the large-scale AudioSet [3] dataset, consisting of approximately 2.1 million weakly labeled videos from YouTube with 527 audio event classes. The backbone CNN contains seven blocks, i.e., blocks B1-B6 and block F1. Each of B1-B5 blocks consists of two convolutional layers followed by a max-pooling layer. B6 consists of one convolutional layer, followed by max-pooling, and F1 consists of one convolutional layer. Batch normalization and a ReLU activation function are applied on the output of each convolutional layer.

To extract compact audio representations, we apply Maximum Activation of Convolution (MAC) on the activations of the intermediate convolutional layers of the feature extraction CNN model [18, 19]. Given a CNN architecture with K convolutional layers, MAC generates K feature vectors $\textbf{h}^{k}\in\mathbb{R}^{C_{k}}$, where $C_{k}$ the number of channels of the $k^{th}$ convolutional layer. The extracted vectors are concatenated in a single feature vector $\textbf{h}\in\mathbb{R}^{C}$, where $C=C_{1}+C_{2}+...+C_{K}$. We applied MAC on the intermediate layers of parts B1-B6 and F1 of the CNN. The dimensionality of the concatenated feature vector amounts to $2,528$ dimensions. Table I presents the dimensionality of the feature vectors extracted from each block of the CNN. Then, we apply PCA whitening[20] to decorrelate the feature vectors. The feature vectors are $\ell^{2}$-normalized before and after the concatenation and also, after the PCA whitening.

Applying $\ell^{2}$-normalization on the extracted feature vectors results in all audio segments having equal contribution to the similarity calculation. This could mean that, for instance, a silent segment would have the same impact as a segment with rich audio content. To overcome this issue, we employ a self-attention mechanism [21] to weigh the audio segments based on their captured information. Given a feature vector h, we use a context vector u to measure its importance. A weight score $a$ is derived by calculating the dot product between the feature vector h and the context vector u. The resulting weight score will be in the range $[-1,1]$ since all vectors have unit norm. However, to avoid the direction change of the feature vectors, we rescale the calculated weight scores in the $[0,1]$ range, by dividing $a$ by $2$ and adding $0.5$. The weighting procedure is formulated in Equation 1.

To calculate video similarity, we first calculate the pairwise similarity matrix that contains the similarity scores between the audio feature vectors of two compared videos. More specifically, given two videos $q$, $p$, with $X$ and $Y$ audio segments respectively, we apply dot product between the feature vectors of the corresponding video descriptors $Q\in\mathbb{R}^{X\times{C}}$ and $P\in\mathbb{R}^{Y\times{C}}$, where $C$ is the dimensionality of feature vectors. This process produces a matrix $\mathcal{S}^{qp}\in\mathbb{R}^{X\times{Y}}$ containing the pairwise similarities between all vectors of the two videos, and can be formulated as a matrix multiplication in Equation 2.

Then, the generated similarity matrix $S^{qp}$ is provided to a four-layer similarity learning CNN network [4]. The network has the capability of capturing the temporal patterns of segment-level within-video similarities. The architecture of the proposed CNN is displayed in Table II. Figure 1 depicts a visual example of the input and the output of the AuSiL CNN. The network can detect temporal patterns and assign high similarity scores in the corresponding segments, i.e., the diagonal part existing in the center of the similarity matrix. At the same time, the noise in the input matrix, introduced by the similarity calculation process, has been significantly reduced in the output. Next, we apply the hard tanh activation function on the network output values to clip them in range $[-1,1]$. The final similarity score is derived by applying Chamfer Similarity (CS), which is formulated as a max operation followed by a mean operation, as in Equation 3.

where $\mathcal{S}_{\upsilon}^{qp}\in\mathbb{R}^{X^{\prime}\times{Y^{\prime}}}$ is the output of the CNN network and Htanh indicates the element-wise hard tanh function.

Ideally, the video similarity score that derives from Equation 3 should be higher for videos that are relevant and lower for irrelevant ones. Therefore, we train our network by organising the training dataset in video triplets $(\upsilon,\upsilon^{+},\upsilon^{-})$, where $\upsilon,\upsilon^{+},\upsilon^{-}$ stand for an anchor, a positive (relevant) and a negative (irrelevant) video respectively. For this purpose, we use the triplet loss function [22], as formulated in Equation 4.

where $\gamma$ is a margin parameter. Triplet loss forces the network to assign higher similarity scores to relevant pairs of video and lower scores to irrelevant ones. Additionally, we employ the similarity regularization loss described in [4], since it provides significant performance improvement. This loss function penalizes the network activations that are out of the clipping range of the hard tanh activation function, as in Equation 5.

The total loss function is defined in Equation 6.

where $r$ is a hyperparameter that determines the contribution of the similarity regularization to the total loss.

Training the architecture described above requires the organisation of the dataset used for training in video triplets. So, we extract pairs of videos with related audio content, to serve as anchor-positive pairs during training. Due to the unavailability of datasets with ground truth annotations in terms of audio content, we extract the positive pairs from a dataset with visual annotations. The videos that have not been labeled as positives are considered negatives. From all positive pairs in terms of visual content, we select only the ones whose global audio feature vectors’ distance is smaller than a certain value. The global audio feature vectors of videos result from the application of global average pooling on the concatenated feature vectors (Section III-A). The upper threshold value was empirically set to $0.175$. We then create video triplets based on the positive pairs by selecting videos that are hard negative examples. More precisely, we select all the anchor-negative pairs whose Euclidean distance in the feature space is less than the distance between the anchor-positive pair plus a margin value $d$, i.e., $D(\upsilon,\upsilon^{-})<D(\upsilon,\upsilon^{+})+d$, where $D(\cdot,\cdot)$ indicates the Euclidean distance between two arbitrary videos. Value $d$ was empirically set to $0.15$.

We employ the VCDB (Video Copy DataBase) [23] to train our AuSiL network. This consists of videos collected from popular video platforms (YouTube and Metacafe) and has been compiled and annotated for the problem of partial copy detection. It contains $528$ videos with $9,236$ copied segments in the core set, and $100,000$ distractor videos in the background set. We use the videos in the core set to form the anchor-positive pairs, and we draw negatives from the background set. A total of 5.8 million triplets is formed from the triplet selection process.

To build an evaluation corpus that simulates DAVR, we employ the FIVR-200K [5] dataset³³3http://ndd.iti.gr/fivr/, which was originally composed for the problem of Fine-grained Incident Video Retrieval (FIVR). It contains 225,960 videos and 100 video queries collected from YouTube based on the major news events from recent years. Table III depicts the annotation labels used in the FIVR-200K, along with their definitions. For the simulation of the DAVR problem, we have set the following annotation procedure. We first select the queries that are suitable for the simulation of the DAVR problem; we excluded 24 queries that were completely silent or noisy, resulting in a set of 76 out of 100 queries. For each of them, we manually annotate the videos with ND, DS, and CS labels according to their audio duplicity with the query. The videos that share audio content with the queries are labeled with the Duplicate Audio (DA) label. In total, we validate 9,345 videos, from which 3,392 are labeled as DA. From this point on, we will refer to this audio-based annotated dataset as FIVR-200K_$\alpha$. Also, for quick comparisons of the different variants of our proposed approach, we sample a subset of the original dataset, which we call FIVR-5K_$\alpha$. For its composition, we first randomly select 50 queries, and then for each one, we randomly draw the 35% of the videos labeled as DA. To make retrieval more challenging, we also add 5,000 distractor videos that are not related to the queries.

To build our second evaluation corpus, we employ the SVD (Short Video Dataset) [6] dataset⁴⁴4https://svdbase.github.io/ that has been composed for the NDVR problem. The dataset consists of over 500,000 short videos collected from a large video platform (TikTok) and includes 1,206 query videos, 34,020 labeled videos, and 526,787 unlabeled videos that are likely not related to the queries. However, due to TikTok’s nature, we empirically found that a large number of audio duplicates exist in the unlabeled set. Therefore, for the composition of an evaluation set that simulates DAVR, we consider only the videos in the labeled set of the SVD dataset. The annotation of the dataset is performed according to the following procedure. We first acquire all the query-candidate video pairs that have been labeled as positives by the original authors of the dataset, and we annotate the video pairs that share audio content. At the end of this process, we discard all queries with no video pairs annotated as positives, resulting in a query set of 167 out of 206 queries. To find potential audio duplicates that are not included in the labeled set, we manually annotate all query-candidate pairs that have not been labeled and have similarity greater than $0.4$. To compute the similarity scores, we follow the process described in Section III-A to extract global feature vectors, and then use the dot product to measure similarity. Based on the described process, we composed an evaluation dataset consisting of 6,118 videos, 167 queries with 1,492 video pairs labeled as audio duplicates. From this point on, we will refer to this audio-based annotated dataset as SVD_$\alpha$.

For the evaluation of retrieval performance, we utilize mean average precision (mAP), which quantifies the ranking of the database items given a query and thus is widely used as a measure for retrieval performance. For the computation of the mAP, we calculate the average precision (AP) for every video query, according to Equation 7.

where $n$ is the number of relevant videos to the query and $r_{i}$ is the rank, based on the similarity score, of the $i$-th retrieved relevant video. The mAP is calculated by averaging the AP scores of all queries.

Additionally, to gain a better understanding about the methods’ performance, we employ the interpolated precision-recall (PR) curve, which shows the trade-off between precision and recall for different thresholds.

To train the network, we employ the Adam optimizer [24] with learning rate $l=10^{-3}$ and regularization parameter $r=0.1$. Also, we consider as the default values of the hyperparameters $t=1s$ and $\gamma=1$. The parameters of PCA whitening are learned from a corpus of one million feature vectors sampled from the VCDB dataset. All experiments were conducted on a machine with Intel Xeon @2.10 GHz CPU and an Nvidia GTX1070 GPU. We trained the network for about 30 hours until convergence. For videos with average duration 100 s, the proposed system needs, on average, 100 ms for feature extraction per video, and 3 ms for the similarity calculation between a video pair.

Initially, we study the impact of time step $t$ on the performance of AuSiL on the subset FIVR-5K_$\alpha$ and SVD_$\alpha$. Table IV illustrates the mAP of the proposed method for different time step values. The time step appears to have a detrimental impact on the system’s performance on SVD_$\alpha$. The smaller time step values report clearly better results compared to the larger ones. Instead, this is not the case for FIVR-5K_$\alpha$, where the selection of the time step seems to have limited impact on the system’s performance. A possible explanation for this could be that SVD mainly consists of short duration videos, i.e., 17 seconds duration on average, unlike FIVR-200K, where the average duration is 117 seconds. Also, using smaller time step values generates larger similarity matrices with richer temporal patterns captured by the AuSiL, leading to more accurate similarity calculation. For the rest sections, we use a time step of 125 ms on the SVD dataset, and a time step of 1 s for all the others.

We also examine the contribution of each AuSiL component. Table V shows the results on FIVR-5K_$\alpha$ and SVD_$\alpha$, first using only the video features extracted from the feature extraction CNN and then adding PCA whitening, the attention mechanism, and the similarity learning CNN. The attention mechanism in every run is trained based on the main training process. Performance improves as individual components are added to the system. The application of PCA whitening has the most significant impact on the network’s performance, with $0.084$ and $0.041$ mAP on FIVR-5K_$\alpha$ and SVD_$\alpha$, respectively. Also, the use of the similarity learning CNN offers a further improvement of $0.052$ and $0.006$ mAP on the corresponding datasets. The contribution of attention mechanism to the overall performance is marginal but positive.

Moreover, we investigate three different settings regarding the transfer and update of the weight parameters of the feature extraction network during training. In the settings where the network weights are updated, we do not use PCA whitening and the attention mechanism, because we encountered network collapse (the network activations were zero for any given input). Table VI presents the results of the three variants on FIVR-5K_$\alpha$ and SVD_$\alpha$. The settings where the parameters are transferred and not updated outperform the other two variants by a considerable margin ($0.794$ and $0.940$ mAP respectively), highlighting that transfer learning was successful. However, the poor performance of the two variants where the weights are updated is noteworthy. A possible explanation for this could be attributed to the different domains represented by the training and evaluation sets, considering that each dataset represents a domain. The network is trained on VCDB; hence, it learns the limited domain represented by this dataset. As a result, the feature extraction CNN fails to transfer knowledge and generalize to the domains of the evaluation sets, and therefore the performance drops. On the other hand, the pre-trained network is trained on AudioSet, a large-scale dataset that represents a very wide domain, and therefore the extracted knowledge can be generalized to datasets of varying domains.

We also investigated the impact of different values of the parameter $\gamma$, as presented in Table VII. The network performs best for $\gamma=1$, achieving $0.794$ and $0.940$ mAP on FIVR-5K_$\alpha$ and SVD_$\alpha$ respectively. For all other gamma values $g=\{0.4,0.6,0.8,1.2\}$, the performance drops significantly.

In this section, we benchmark our method based on the FIVR-200K_$\alpha$ and SVD_$\alpha$ datasets that we developed to simulate the DAVR problem. To evaluate the proposed AuSiL approach, we have re-implemented two state-of-the-art methods proposed for the CBCD problem [14, 16]. These methods are based on binary images generated by the spectrogram of the audio signal. We will refer to them as Spectro Slides [14] and Spectro Tiles [16]. Also, we compare against the publicly available open-source framework Dejavu [17], a Shazam implementation for audio recognition.

Table VIII illustrates the performance of the compared approaches on the two datasets. On FIVR-200K_$\alpha$, the proposed approach reaches $0.701$ mAP, significantly outperforming the Spectro Slides and Tiles systems by $0.113$ mAP and $0.191$ mAP respectively; the Dejavu system surpasses the performance of AuSiL, achieving the best score of $0.726$ mAP. AuSiL achieves the best performance on SVD_$\alpha$ with $0.940$ mAP, outperforming all three competing methods by a significant margin, i.e., $0.066$ mAP from the second-best method, Dejavu. Looking for a reasonable explanation for the performance difference on the two evaluation datasets, we have empirically determined that a considerable amount of audio duplicates in the SVD_$\alpha$ have been created using speed transformations on the audio signals. AuSiL is robust to such transformations due to its deep learning components that are trained to handle such variance. On the other hand, the other three methods rely on handcrafted methods and, therefore, can only tackle a limited range of such transformations. We set up an evaluation scheme in the next subsection in order to validate this hypothesis. Nevertheless, we experimentally found that all approaches fail to retrieve the following cases: i) the duplicate audio signal has been mixed with high volume irrelevant audio signals, e.g., speech or music, ii) the duplicate segment is too short (up to 1 second), and iii) the duplicate audio signal is very distorted.

Figure 2 presents the Precision-Recall curves of the competing approaches on FIVR-200K_$\alpha$ and SVD_$\alpha$, respectively. In the case of FIVR-200K_$\alpha$, AuSiL’s curve is below the Dejavu’s curve up until 0.6 recall, outperforming or reporting similar performance on the remaining recall points. Additionally, AuSiL’s curve is above those of the Spectro methods, except for the initial recall points. In the case of SVD_$\alpha$, AuSiL’s curve lies above all other curves with a clear margin.

To delve into the performance of the competing methods, we set up an evaluation scheme to benchmark the robustness of our approach to audio speed transformations. We test the FIVR-200K_$\alpha$ and SVD_$\alpha$ according to the following procedure. We first employ the dataset queries to artificially generate audio duplicates by applying speed transformations on the audio signals. We use the following factors for the generation: $\{\times 0.25,\times 0.5,\times 0.75\}$ for slow down and $\{\times 1.25,\times 1.5,\times 2\}$ for speed up. Then, we exclude from the datasets all videos that are originally labeled as audio duplicate, and we only consider as positives the artificially generated audio duplicates.

Our proposed method proves to be very robust on speed transformations, reaching a performance of $0.865$ and $0.923$ mAP on FIVR-200K_$\alpha$ and SVD_$\alpha$, respectively. On the other hand, Dejavu, the best performing method on FIVR-200K_$\alpha$, performs poorly, achieving only $0.443$ and $0.741$ mAP respectively. The Spectro Slides and Tiles methods do not work at all on this setup, reporting near-zero mAP. This highlights that the proposed approach tackles the limitation of the previous methods and robustly calculates the similarity between audio duplicates generated from speed transformations.

Finally, we evaluate the performance of AuSiL and the competing approaches on the much more challenging setting of visual-based video retrieval. Although these tasks are not designed for benchmarking audio-based methods, they can still provide an indication of retrieval performance. We use the two original datasets presented in Section IV-A, i.e., FIVR-200K and SVD, and also the EVVE (EVent VidEo) [25] dataset that is designed for the event-based video retrieval problem. The FIVR-200K consists of three different evaluation tasks simulating different retrieval scenarios: i) the Duplicate Scene Video Retrieval (DSVR), ii) the Complementary Scene Video Retrieval (CSVR), and iii) the Incident Scene Video Retrieval (ISVR). As expected, the performance of audio-based approaches is far worse compared with the visual-based ones, due to the fact that the visual relevance of two videos does not imply that they are also related in terms of audio.

Table IX presents the performance of the audio-based approaches on the FIVR-200K [5], SVD [6] and EVVE [25] datasets. Additionally, the table depicts the best state-of-the-art visual-based methods in each case, i.e. ViSiL [4] for FIVR-200K and EVVE, and DML [26] for SVD. On FIVR-200K, AuSiL is outperformed by Dejavu on DSVR and CSVR tasks, but it achieves the best performance on the ISVR task with $0.232$ mAP. On SVD, AuSiL outperforms the competing audio-based approaches, achieving $0.516$ mAP and surpassing the second-best approach, Dejavu, by $0.039$ mAP. On EVVE, our approach achieves $0.288$ mAP, significantly higher than all three competing methods, with the second one reporting $0.160$ mAP. As expected, in all evaluation cases, there is a large gap in relation to the performance of the state-of-the-art visual-based approaches.