DSCLAP: Domain-Specific Contrastive Language-Audio Pre-Training

For encoding audio frames we utilize a pre-trained Wav2vec2 whose architecture follows the work in [9]. Let the raw waveform be $\textbf{\footnotesize{X}}_{a}$ and $\mathcal{F}_{a}$ represent the Wav2vec2 encoder. For $N$ waveforms in a batch, the final acoustic representations are calculated as follows:

where $h^{(i)}_{a}\in\mathbb{R}^{d}$ is the $i$-th audio representation of dimensionality ${d}$. A $Pooling$ layer ($\mathit{i.e.,}$ mean-pooling) is applied to aggregate the frame-level features into an utterance-level representation.

We choose TinyBERT [12] as the text backbone network, a Transformer model trained with attention-based distillation and hidden states-based distillation. Let the text be $\textbf{\footnotesize{X}}_{t}$ and $\mathcal{F}_{t}$ represent the TinyBERT encoder. Similar to the audio encoding process, we can obtain the linguistic representation $h^{(i)}_{t}$:

Given a representation pair ($h^{(i)}_{a}$ , $h^{(i)}_{t}$), following CLIP [13], we use InfoNCE loss to measure dependencies between the audio and text modality. Specifically, we project $h^{(i)}_{a}$ and $h^{(i)}_{t}$ onto the latent embedding space. Then we maximize the similarity between the pair of audio and text representation ($\mathit{i.e.,}$ positive pair) while minimizing the similarity between the negative pairs as follows:

where $\phi_{a}$ and $\phi_{t}$ are learned fully-connected projection functions, $\textbf{z}^{(i)}_{a}$ and $\textbf{z}^{(i)}_{t}$ are hidden representations of the $d$ dimension, $sim()$ is a similarity function ($\mathit{e.g.,}$ dot product). Furthermore, $\mathcal{B}=\{\textbf{z}^{(1)}_{\beta},\textbf{z}^{(2)}_{\beta},...,\textbf{z}^{(N)}_{\beta}\}$ ($\beta\in\{a,t\}$) is a set of hidden representations, which contains a positive sample $\textbf{z}^{(i)}_{\beta}$ and N-1 negative samples ( $\mathcal{B}-\textbf{z}^{(i)}_{\beta}$).

Contrasting learning objective often requires a large batch size, which in turn increases a tremendous computational resource cost. To cut down the heavy resource dependency, we introduce a language-audio matching objective with hard negative sampling [16] to allow us to conduct efficient multimodal representation alignment with limited resources. Specifically, for a given fixed multimodal representation pairs ($\textbf{z}^{(i)}_{a}$ , $\textbf{z}^{(i)}_{t}$), a set of hard negative examples { ($\textbf{z}^{(i)}_{a}$ , $\textbf{z}^{(k)}_{t}$), ($\textbf{z}^{(k)}_{a}$ , $\textbf{z}^{(i)}_{t}$)}, are sampled from the top-K most similar representations in the training batch. In other words, we use carefully selected hard negative samples as an additional negative sample for contrastive learning as follows:

Incorporating the loss on the contrastive learning and the cross-modal matching introduced above, we estimate the parameters of the pre-train model $\theta$ by minimizing the following objective:

where $\lambda$ and $\gamma$ are hyperparameters that determine the contribution of each regularization component. For all the experiments, we set $\lambda$ and $\gamma$ as $0.5$, which we searched through cross-validation.

We use AdamW optimizer for pre-training and the learning rate is initialized to 2e-5. The the temperature $\tau$ of contrastive loss is initialized to $log(1\//0.07)$. The maximum sequence length is set to 16 and 80,000 for texts and audio waveforms respectively. Our model is implemented in PyTorch Lightning and transformers libraries. We train it for 20 epochs with mixed precision, on 8 NVIDIA RTX 3090 GPUs with a batch size of 64 per GPU. The whole training process takes 7.5 days to complete. Notably, for a fair comparison, we used five random seeds (i.e. 1, 12, 123, 1234, and 12345) to trigger experiments in both of the downstream tasks and report the average performance of the five experiments for each task.

We combined several recent state-of-the-art unimodal pre-train models to benchmark their performance on domain-specific downstream tasks and report the results of the two best-performing models: (Whisper [10], BERT [12]) and (Wav2vec2 [9], BERT [12]), where the better-performing (Wav2vec2, BERT) is set as a baseline backbone network (annotated by Base). Additionally, we also introduce two novel competitive optimization methods (Rdrop [18] and OGM-GE [19]) on the Base model.

The comparative results are presented in Table 2. In both datasets, DSCLAP achieves the best performance and surpasses the variants across all metrics (MDSD and MCIC combined). This confirms that our method can be better suited to domain-specific multimodal tasks. We observe that the proposed model significantly reduced the false rejection rate (FRR) in the MDSD task from 5.35% to 2.77%, which is of great benefit to IVAs because it will enable IVA to respond to more user requests without wake-up steps.