Improving CTC-based ASR Models with Gated Interlayer Collaboration

Transformer encoder. An encoder layer of the Transformer [6] comprises two main sub-layers: a self-attention layer followed by a feed-forward network. Formally,

$$\begin{split}{\rm h}_{l}^{\prime}&={\rm h}_{l}^{in}+{\rm SAN}({\rm Q}_{l},{\rm K}_{l},{\rm V}_{l}),\\ {\rm h}_{l+1}^{in}&={\rm h}_{l}={\rm h}_{l}^{\prime}+{\rm FFN}({\rm LN}({\rm h}_{l}^{\prime})),\end{split}$$

(1)

where ${\rm h}_{l}^{in}$ and ${\rm h}_{l}$ are the input and output of $l$-th layer. $\rm LN(\cdot)$, $\rm SAN(\cdot)$ and $\rm FFN(\cdot)$ are layer normalization [26], attention mechanism, and feed-forward networks respectively. (${\rm Q}_{l}$, ${\rm K}_{l}$, ${\rm V}_{l}$) are query, key and value vectors that are transformed from the normalized $l$-th encoder layer input, that is ${\rm LN}({\rm h}_{l}^{in})$.

Conformer encoder. Conformer [25] combines Transformer and convolutional neural network layers to efficiently learn both global and local representations. An encoder block of Conformer is composed of four modules, i.e., a feed-forward module, a self-attention module, a convolution module, and a second feed-forward module. Formally,

$$\begin{split}{\rm h}_{l}^{\prime}&={\rm h}_{l}^{in}+\frac{1}{2}{\rm FFN}({\rm h}_{l}^{in}),\\ {\rm s}_{l}&={\rm h}_{l}^{\prime}+{\rm SAN}({\rm Q}_{l},{\rm K}_{l},{\rm V}_{l}),\\ {\rm o}_{l}&={\rm s}_{l}+{\rm Convolution}({\rm s}_{l}),\\ {\rm h}_{l+1}^{in}&={\rm h}_{l}={\rm LN}({\rm o}_{l}+\frac{1}{2}{\rm FFN}({\rm o}_{l})).\end{split}$$

(2)

Connectionist Temporal Classification. CTC [11] optimizes the model by maximizing the likelihood of all valid frame-wise alignments $\beta^{-1}(Y)$. Formally,

$$\begin{split}{\rm q}_{L}&={\rm Softmax}({\rm Linear}_{d_{m}\rightarrow{|V|}}({\rm LN}({\rm h}_{L}))),\\ P_{ctc}(Y|{\rm h}_{L})&=\sum_{\alpha\in{\beta^{-1}(Y)}}\prod_{t=1}^{T}{\rm q}_{L}^{t}[\alpha_{t}],\end{split}$$

(3)

where ${\rm q}_{L}$ is the probability distribution of each frame over the vocabulary $V$, and $d_{m}$ is the dimension of the attention layer. ${\rm q}_{L}^{t}$ denotes the $t$-th column of ${\rm q}_{L}$, and ${\rm q}_{L}^{t}[\alpha_{t}]$ denotes the $\alpha_{t}$-th symbol of ${\rm q}_{L}^{t}$. ${\rm h}_{L}$ is the output of the encoder module, and $L$ indicates the number of encoder layers. The CTC loss is defined as the negative log-likelihood as:

$$\pounds_{ctc}=-\log\ P_{ctc}(Y|{\rm h}_{L}).$$

(4)

Intermediate CTC. Intermediate CTC [21] utilizes the representations of intermediate encoder layers to calculate the auxiliary CTC losses to further improve the performance of the models. We train the model with a total of $K$ intermediate layers to calculate the inter-layer CTC losses. The total loss function is a linear combination of the final layer CTC loss $\pounds_{ctc}$ (Eq.4) and the intermediate CTC losses. Formally,

$$\pounds=(1-{\lambda})\pounds_{ctc}+{\lambda}\frac{1}{K}\sum_{i=1}^{K}\pounds_{\frac{iL}{K+1}}.$$

(5)

The intermediate CTC loss $\pounds_{l}$ is calculated with the output of the $l$-th encoder layer (${\rm h}_{l}$) as

$$\pounds_{l}=-\log\ P_{ctc}(Y|{\rm h}_{l}).$$

(6)

Figure 1 shows the framework of the proposed method. We propose a gated interlayer collaboration (GIC) mechanism to make CTC-based models have access to textual information generating by intermediate predictions of the encoder module. The GIC block first introduces an $embedding$ layer that is used to get the position-specific textual representation by a weighted-combination of all token embeddings. Formally,

$${\rm e}_{l}^{j}=\sum_{i}^{|V|}{\rm q}_{l}^{(i,j)}\odot emb_{i},$$

(7)

where $emb_{i}$ represents the $i$-th column of the embedding matrix. ${\rm q}_{l}^{(i,j)}$ indicates the $i$-th probability mass in the soft labels at the $j$-th frame, where the soft label is calculated by a softmax activation given the hidden state ${\rm h}_{l}^{j}$ of the $j$-th frame

$${\rm q}_{l}={\rm Softmax}({\rm Linear}_{d_{m}\rightarrow{|V|}}({\rm LN}({\rm h}_{l}))).$$

(8)

In Eq.8, $l$ is one of the particular intermediate layers and the output of the $l$-th layer ${\rm h}_{l}$ is used to calculate the intermediate CTC loss in Eq.6. Then, a gate unit combines the acoustic features ${\rm h}_{l}$ and the textual features ${\rm e}_{l}$ by a $sigmoid$ activation:

$$\begin{split}{\rm g}_{l}&={\rm Sigmoid}(W_{1}{\rm h}_{l}+W_{2}{\rm e}_{l}+b),\\ {\rm h}_{l+1}^{in}&={\rm g}_{l}\odot{\rm h}_{l}+(1-{\rm g}_{l})\odot{\rm e}_{l}.\end{split}$$

(9)

The gated combination ${\rm h}_{l+1}^{in}$ is then fed into the next encoder layer, and thus our approach benefits from both the textual and the acoustic interactions.

We conducted experiments using ESPnet [33]. We adopt two CNN-based downsampling layers followed by an 18-layer encoder for our experiments. For both Transformer and Conformer backbones, the dimension of the attention layer is 256 with 4 split heads, and the dimension of the feed-forward layer is 2048. Specifically, the kernel size is 15 for Conformer. We use a single-LSTM-layer decoder for the transducer-based models with the same 18-layer encoder. The Adam optimizer [34] with 25000 warmup steps are used for training. We train the models for 100 epochs on both AISHELL-1 and TEDLIUM2, and 50 epochs on AIDATATANG. We conduct experiments to compare our method with previous CTC-based models under the same model scale, including CTC [11], intermediate-CTC training with the intermediate CTC loss [21], and SC-CTC training with the intermediate CTC loss and the self-conditioning mechanism [22]. The results are obtained by either CTC greedy decoding without the external LM or beam search (beam size = 10) with a 4-gram LM.

Figure 2 shows the effect of different numbers of $K$ and different values of $\lambda$ in Eq.5. We found that $K=5$ and $\lambda=0.5$ achieve better results. Therefore, we set $K=5$ (interlayer set of $\left\{3,6,9,12,15\right\}$) and $\lambda=0.5$ for our experiments.

Table 1 shows the results of different methods on AISHELL-1 based on the Conformer backbone. In addition to recently published CTC models [24], we compare our method with a few state-of-the-art models, including the hybrid CTC/Attention model [29] and the transducer-based method [30]. Our model sets the start-of-the-art results among all Conformer-based models. Specifically, our method achieves CER of 4.0%/4.4% on AISHELL-1 dev/test sets using CTC greedy decoding without the external LM, outperforming the hybrid systems and other CTC models with slightly overhead parameters, and outperforming the transducer-based models while having the benefit of non-autoregressive decoding manner. Furthermore, our method performs better to alleviate the conditional independence hypothesis of CTC-based models compared with the LM shallow fusion and the transducer network. Our method achieves CER of 4.0%/4.3% on AISHELL-1 dev/test sets with a 4-gram LM for shallow fusion. The external LM brings less boost since the proposed method introduced text interactions in the training process.

Table 2 and Table 3 show the results of different methods with the Conformer backbone on TEDLIUM2 and AIDATATANG, respectively. GIC achieves WER of 7.9%/7.3% with the CTC greedy search method and WER of 7.7%/7.1% with a 4-gram LM for shallow fusion on TEDLIUM2 dev/test sets, outperforming the results in recent literature [31, 23, 32]. The results show that GIC is also effective on the English dataset. Moreover, GIC achieves CER of 3.8%/4.4% on AIDATATANG dev/test sets without the external LM.

Table 4 shows the results on AISHELL-1 and AIDATA-TANG datasets, using the Transformer backbone. Our method is also effective for the Transformer network and achieves the lowest CER (i.e., 5.1% on the AISHELL-1 test set without LM) among all other Transformer-based models.

The proposed approach differs from the standard CTC [11] model in a number of ways, including the GIC block and intermediate CTC losses. We study the effect of these differences by mutating the proposed approach toward the CTC model. Table 5 shows the impact of each change on the proposed method. The results demonstrate that introducing the GIC mechanism brings performance improvements from 5.1%/5.9% to 4.8%/5.5% on AIDATATANG dev/test sets.

Figure 3 gives the CER comparison of all interlayer predictions of the SC-CTC and GIC methods. GIC achieves better performance in all intermediate layers as well as the final layer. Compared with the self-conditioning mechanism [22], the GIC mechanism performs better to improve the performance of the CTC-based model.