ROYALFLUSH SPEAKER DIARIZATION SYSTEM FOR icassp 2022 MULTI-CHANNEL MULTI-PARTY MEETING TRANSCRIPTION CHALLENGE

The schematic of multi-channel U-Net-based architecture is shown in Fig.2, which is obtained by following [19]. The U-Net-based architecture has three paths: the downsampling path, connection path and upsampling path. The downsampling path consists of three Squeeze-and-Excitation(SE) Conv blocks[20] involving Conv block and SE module. Conv block includes a convolutional layer, batch normalization(BN) and Rectified Linear Unit(ReLU). Note that the first SE Conv block does not downsample feature maps, the second and third downsample feature maps to half the size of the input, respectively. The connection path contains 9 repeated Residual blocks[21] with constant feature map size. The upsampling path and downsampling path are symmetric and comprises three SE Conv blocks, where deconvolution is used to enlarge the feature map size. There are skip connections between the downsampling path and the upsampling path, and the corresponding feature maps are summed up as the input of the SE Conv block of the upsampling path. Finally, the feature maps generated by upsampling path are flattened along the channel and frequency dimensions, and fed to a fully connected(FC) layer. The detailed parameters of the architecture are shown in Table1.

We experiment with two types of U-Net-based architecture. The first type is shown as described above, and it is called SEUnet1. Multi-channel 2D features are used as the input to the SEUnet1. Since the 3D convolutional layers are used, the computational complexity of the neural network will be greatly increased. The second type of the U-Net-based architecture is a mix of 2D convolutional layers and 3D convolutional layers. We just added a new 3D convolutional layer to the first layer of the SEUnet1 architecture with a kernel size of 8 × 1 × 1 and a stride of 1, which we call SEUnet2.

The input features are 64-dimensional log Mel-filterbank energies with a frame length of 25ms and a frame shift of 10ms. Utterance-level Cepstral Mean Normalization (CMN) by subtracting the cepstral mean from the whole utterance is applied to all input features. The log Mel-filterbank energies from each channel of multi-channel audio are stacked together as input to the neural network. As a data augmentation step, we randomly mask the Mel-filterbank energies 0 to 10 bins in the frequency dimension.

The models are trained to perform classification for three classes: silence, single speaker speech and overlap speech. The training sample length is 4 seconds(400 frames), and the mini-batch size during training is 32. The stochastic gradient descent (SGD) optimizer with weight decay 2e-5 and the softmax cross-entropy loss are employed. The initial learning rate is 0.01 and is multiplied by 0.9 after each epoch.

We take the posterior probability of overlap speech node in the output layer to determine whether it is overlapped speech frame by frame. Inference is performed over length of 400 frames with a stride of 200 frames. The outputs on overlapped segments are averaged to obtain the final posterior probability. We set the threshold to 0.55, which is tuned on the far-field Alimeeting evaluation set. In order to reduce the false alarm of OSD, we refine the OSD results using oracle VAD labels.

For comparison, we also conduct experiments in a single-channel setting. We re-implemented the BLSTM architecture that is used for OSD as our baseline[22]. It has two layers of 128 bidirectional LSTM units with a projection layer, followed by an FC layer with 128 neurons. The results of SEUnet1 on single-channel speech input are also reported in this paper. All training configurations are the same as described above, except that the input is single-channel speech.

OSD model’s performance is using frame-level precision, recall and F1 score. Table 2 and Table 3 shows the performance of the multi-channel models and single-channel models on the far-field Alimeeting evaluation set respectively. As shown in Table 3, the proposed SEUnet1 architecture brings an absolute F1 improvement of 4.29% over the baseline, and as in Table 2, the additional improvement using multi-channel training translates to an absolute F1 improvement of 5.26%. Finally, the equal-weighted fusion of the posterior probabilities of SEUnet1 and SEUnet2 brings another 0.31% absolute F1 improvement. Compared to baseline, F1 has significantly improved from 74.18% to 84.04%. In subsequent experiments, we use the fusion of SEUnet1 and SEUnet2 as the OSD results.

The TDNN model that we used is based on x-vector topology. The original x-vector topology includes five TDNN layers that learn frame-level representations from acoustic features, a statistical pooling layer that combines mean and standard deviation of the frame-level representations and two FC layers. [26] has shown that the simple temporal standard deviation pooling(TSDP) can perform better than statistical pooling. Therefore, we replace the statistical pooling of the standard x-vector topology with TSDP, we called TDNN-TSDP in this paper. The speaker embedding is extracted from the first FC layer.

The Resnet34 is another mainstream architecture in the speaker verification task, also called r-vector topology. It comprises of 34 stacked Residual blocks, with a statistical pooling layer and an FC layer. We use the TSDP version, which we call Resnet34-TSDP, in this paper. More details can be found in [26].

The ECAPA-TDNN provides state-of-the-art speaker recognition performance through a careful architecture design. It introduces several enhancements methods based on the x-vector topology, including 1-dimensional SE-Res2Block, multi-layer aggregation, and channel-dependent attentive statistical pooling. We use a large version of the original paper for this model, with 1024 filters. More details can be found in [25].

All speaker embedding models are trained using AAM-softmax[27] with a margin of 0.15 and a scale of 32. The input features are 64-dimensional log Mel-filterbank energies processed with utterance-level CMN, the same as for training the OSD model. The training data is cut into 1.5-second length segments, and the mini-batch size for training is 128. The SGD optimizer with weight decay 2e-5 is employed. The initial learning rate is 0.1 and is multiplied by 0.9 after each epoch.

Table 4 shows the equal error rate(EER) of the speaker embedding models on the CNCeleb-E test set, and the best one is the ECAPA-TDNN. The performance of ECAPA-TDNN in the third row is 8.82% EER. As far as we know, it is the best performance publicly reported.

To determine the speaker number in the diarization process, we utilized the prior knowledge about the data set. Since the OSD model is not perfect, some overlapped segments will inevitably deteriorate the speaker embeddings, and lead to the wrong speaker number estimation of the clustering. In this work, we use the information that the maximum number of speakers is four in the Alimeeting corpus. The overlap ratio of each session is calculated by the OSD model. If the overlap ratio is greater than 20%, we consider the number of speakers in the session to be four. The motivation to do that is based on the consideration that the higher the proportion of overlapped speech segments, the more speaker in the meeting.