MLNET: An Adaptive Multiple Receptive-field Attention Neural Network for Voice Activity Detection

The MLNET-based VAD achieved a frame-based speech or non-speech classifier. Suppose the training corpus can be marked as $\chi={\{(x_{t},y_{t})\}}_{t=1}^{T}$ , where $x_{t}$ is an acoustic feature vector of $t$-th frame audio signal and $y_{t}$ denotes the label of $x_{t}$. If $x_{t}$ is a speech frame, then $y_{t}=1$; otherwise, $y_{t}=0$. Because of contextual information’s importance for speech applications, $x_{t}$ is usually expanded to $I_{t}=[x_{t-r},...,x_{t-1},x_{t},x_{t+1},...,x_{t+r}]$ when training or testing. Here, the objective of VAD is to learn a function $f$ as (1).

$$\hat{y_{t}}=f([x_{t-r},...,x_{t-1},x_{t},x_{t+1},...,x_{t+r}])$$

(1)

where $\hat{y_{t}}$ denotes the predicting result of $x_{t}$ and $r$ denoted the window size of contextal speech information.

In experiment, the value $r$ was a hyper-parameter and the fixed value of $r$ was hard to adapt various speech environments. For example, when $x_{t}$ wa a speech frame and the speech segment duration around $x_{t}$ was short, the large $r$ would contain much non-speech information and cause false positive results. In turn, when $r$ was short, the contextual information may not be fully utilized, which can result in false negative or false positive results. To address the above problems, this paper proposed the MLNET model, which leveraged the multiple gated affined units and attention mechanism to choose the optimal receptive-field speech adaptively to make the classification. The MLNET’s architecture is shown in Figure 1.

The detail architecture of multiple receptive-field attention block was showed in Figure 2. In order to realize adaptive feature selection, multiple branches were leveraged to extract different receptive-filed speech features and each branch represents a specific receptive field or a specific contextual information. Specifically, for the branch of $r_{i}$, the contextual speech information can be denoted as $I_{t}^{{}^{\prime}}=[x_{t-r_{i}},...,x_{t-1},x_{t},x_{t+1},...,x_{t+r_{i}}]$ and $(2\times r_{i}+1)$ frame features were included. For the calculation of subsequent attention modules, the feature matrix of $I_{t}^{{}^{\prime}}$ with different size need to be converted feature matrix with the same size. This paper made use of the gated affined unit to finish this task. The gated affined unit as this non-linearity has proved to work better for modeling audio than the other affine function or Relu functions [26]. The gated affined unit’s definition was defined as (3):

$$q_{i}=tanh(W_{f,r_{i}}*I_{i}+b_{f,r_{i}})\odot\sigma(W_{g,r_{i}}*I_{i}+b_{g,r_{i}})$$

(2)

where $W_{f,r_{i}}$, $b_{f,r_{i}}$ denoted affine parameters of tanh and $W_{g,r_{i}}$, $b_{g,r_{i}}$ denoted affine parameters of sigmoid. It was obvious that the size of $W_{f,r_{i}}$ and $W_{g,r_{i}}$ would change with $r_{i}$ while the size of $q_{i}$ was constant regardless of $r_{i}$. The $*$ denoted a convolution operator while the $\odot$ denoted an element-wise multiplication operator.

After gated affined unit operation, different two-dimensional extracted matrixes, which were denoted by $q_{t,1},...,q_{t,i},...,q_{t,r}$, were obtained and each feature matrix of $q_{t,i}$ had the same size. Analogy with image, each feature matrix of $q_{t,i}$ can be regarded as a channel feature map and we produced a channel attention map to decide which feature matrix should be focused on. Based on previous channel attention work in images [27, 28], the attention module’s structure was showed in Figure 3. Firstly, both average-pooling and max-pooling operations were used to aggregate information of a feature matrix and two feature descriptors were generated to represent different feature matrixes. Then, both descriptors were forward to a shared 2-layer fully connected DNN to produce two different attention ratio vector $p_{t,max}$ and $p_{t,avg}$. Finally, by adding an norm operation, $p_{t,max}$ and $p_{t,avg}$ were used to generate final attention ratio vector $a_{t}$ of (4).

	$\displaystyle a_{t}$	$\displaystyle=\sigma{(FC(avgpool(q_{t}))+FC(maxpool(q_{t})))}$		(3)
		$\displaystyle=\sigma{(W_{1}(W_{0}(avgpool(q_{t})))+W_{1}(W_{0}(maxpool(q_{t}))))}$

where $FC$ denoted the shared 2-layer fully connected DNN and $W_{1}$ and $W_{0}$ were the weights of DNN. Note that the weight $W_{0}$ of first layer followed a $leaky\_relu$ activation function. After obtaining $a_{t}$, the attentioned or scaled feature matrix would be calculated through (4)-(7).

$\displaystyle a_{t}$

$\displaystyle=[a_{t,1},...,a_{t,i},...,a_{t,r}]$

(4)

$$p_{t,i}=\sigma{(a_{t,i})}/(\sum_{i=1}^{r}{\sigma{(a_{t,i})}})$$

(5)

$$p_{t}=[p_{t,1},p_{t,2},...,p_{t,r}]$$

(6)

$$Q_{t}=\sum_{i=1}^{r}{p_{t,i}*q_{t,i}}$$

(7)

Where $a_{t}$ denoted the attention vector and it can be calucated by the attention module. $\sigma$ was the sigmoid function and the $p_{t}$ represented the normalized value of $a_{t}$. Finally, the $Q_{t}$ was calculated by summing original $q_{t}$ with attentional coefficient.

Through the adaptive multiple receptive-field attention block, the scaled feature matrix $Q_{t}$ was obtained and $Q_{t}$ would be reshaped into a feature vector $m_{t}$. The Bi-LSTM is skilled in learning contextual speech information and the DNN is an excellent classifier. Nextly, the feature sequence $M_{t}=[m_{1},m_{2},...,m_{T}]$ would be fed into a two-layer Bi-LSTM and 1-layer Fully Connected DNN to make the final classification.

The training of MLNET-based VAD could be regarded as a common supervised optimized problems with traditional cross-entropy loss function.

$$\mathcal{L}_{crossentropy}=\sum_{t=0}^{t=T}{\{y_{t}log(\hat{y}_{t})+(1-y_{t})log(1-\hat{y}_{t})\}}$$

(8)

Because of the characteristic of the proposed model, this paper further investigated an additional attention loss function to adapt the attention mechanism in the training phase. The attention loss function was designed to emphasize the most important receptive field of $r_{k}$, and the definitions were showed in (9)-(11).

$$k=\underset{i}{argmax}(p_{t,i})\\$$

(9)

$$\mathcal{L}_{attention}=\sum_{t=0}^{t=T}\sum_{i=0}^{i=r}(y_{t,i}log(p_{t,i}))$$

(10)

$$\mathcal{L}=\mathcal{L}_{crossentropy}+\mathcal{L}_{attention}$$

(11)

Where $\mathcal{L}_{attention}$ could be denoted as a softmax problem and the most appropriate receptive-field $p_{t,k}$ was the target and the corresponding $y_{t,k}$ is $1$ and other $y_{t,i}=0({i\not=k})$.

In this paper, the English corpus of Aurora4 [29] and the Chinese corpuses of Thchs30 [30] were applied to train and test the proposed model. In our experiments, firstly, because of imbalance of speech and non-speech, 2-second-long silence segments were added into forward and backward of each utterance. In training, the clean speech corpus were corrupted by public 100 noise types of HuNonspeech¹¹1http://web.cse.ohio-state.edu/pnl/corpus/HuNonspeech/. Each utterance was randomly corrupted at a level of -5dB-20dB SNR and all have an average 7.5dB SNR. The NOISEX-92²²2http://spib.rice.edu/spib/select_noise.html noise dataset was used to construct testing dataset and 4 types unseen noises of babble, factory, destory-engine were selected to corrupt the clean speech. Also, the SNR was setted between -5dB and 20dB and the average is 7.5dB. For Aurora4, 95% of training data were used for training and 5% were used as dev data. The testing corpus of Aurora4 were corrupted by the NOISEX-92 and leveraged as the testing data. For thchs30 data, the dataset was constructed with the same process, but the different was that dev dataset leveraged the original corresponding data.

For comparison, the metrics of F1-score and DCF were selected as a performance measurement. F1-score took into account both accuracy and recall metrics, which was a common evaluation index of binary classification problems. DCF³³3http://fearlesssteps.exploreapollo.org/ reflected the wrong performance of the model and DCF was defined as followed:

$$F1-score(\theta)=\frac{2TP}{2TP+FP+FN}$$

(12)

$$DCF(\theta)=0.75\times P_{FN}(\theta)+0.25\times P_{FP}(\theta)$$

(13)

where $\theta$ denoted a given system decision-threshold setting. TP represented true positive examples’ num while FP and FN represented the num of false positive and false negative examples. $P_{FN}$ was the probability of FP while $P_{FN}$ was the probability of FN. It should be noted that the larger the F1-score was, the better performance while the smaller the DCF was, the better performance. In testing, we calculated the two metrics of each recording respectively and averaged the metrics of all recordings as the final score.

The acoustic feature extracted for MLNET was 40-dimensional log-mel filterbank while the frame size was 25 ms with a shift of 10 ms. The window of attention block were setted as 19 frames, which corresponded to 190ms contextual information, while the gated affined unit’s receptive-field were setted 1, 3, 5, 7 ,9. Other parameters of MLNET were shown in Table1. For training our proposed model, the matrix weight parameters of MLNET were all initialized with random uniform initialization while the bias parameters were initialized with a constant of 0.1. In our experiments, we trained the network for 150 epoches with the Adam algorithm when the loss function got little change. The batchsize was 32 and the learning rate was set to 0.001. When training, the gradient cropping strategy was also applied and the gradient of each parameter at each iteration was limited between -1 and 1.

To demonstrate the effectiveness of our proposed model, three VAD approaches were used for comparison. The first was google’ WebRTC VAD systems [31]. Additionally, Vafeiadis proposed 2-D CRNN model and has made great success in recent speech activity detection [32]. Nextly, ACAM approach, which was the state of art of attention-based methods, was also included [23]. In our experiments, we made use of the same parameters, but all trick strategies, such as batch normalization and regularization, were not leveraged. To alleviate the effect of the input features, it should be noted that 40-log mel acoustic features were leveraged to establish the CRNN and ACAM baseline models, which were different from the original approaches.

The results were summarized in Table 2 and Table 3. We can observed that MLNET has the best performance and outformed other three baselines models. Especially, three-training-based methods achieved 10% higher than google’s VAD in F1-score and 8% lower in DCF. The ACAM and MLNET outformed the traditional deep learning methods of CRNN, which proved attention-based models were helpful for improving the accuracy of detection. Comparing with ACAM, MLNET selected the windows’ attentions instead of ACAM’s frames-based attention and experiments showed that the window based attention models achieved higher performance than the frames-based models, which also conformed to our prior knowledge that current frame information was the most important and the joined frames were auxiliary information when predicting the current frame.

In order to illustrate each part’s functionality of our proposed model, the comparison of each modules were further investigated. The base was the Bi-LSTM, which just leveraged the contextual speech information and the feature vectors of contextual speech were aggregated to a longer vector before feeding into the network. The second was the gated unit model that the gated unit operation replaced the aggregated mechanism. The third and the fourth were leveraged to certificate the multiple window’s functions while the attention’s function was also compared. The Aurora4 dataset were leveraged to make this evaluation and the results were shown in Table 4. As noted in this table, the gated affined unit based models have better performance than direct aggregation of Bi-LSTM and achieved about 2% increase in F1-score and 1.5% decrease in DCF. Adding the multiple receptive-field of non-attention, the VAD’s performance was also improved and achieved about 2.5% increase in F1-score and 2.13 % decrease in DCF. Lastly, the adaptive attention mechanism was also helpful for MLNET’s performance. In contrast, the adaptive attention only achieved a small accuracy improvement than non-attention module and the reason may be the mechanism of receptive-field selection has existed in multiple receptive-field attention block. In particular, we observed that the adaptive multiple receptive-field attention block was also helpful for speeding up models’ convergence in our experiments. To sum up, the proposed method can better deal with the VAD problems.