Adaptive Convolution for CNN-based Speech Enhancement Models

DyConv [21] is an effective method to enhance the representation capability of models with negligible additional computational complexity. It employs input-dependent dynamic kernels instead of static counterparts, making the kernels nonlinear functions of the input features. The dynamic kernels are generated by aggregating a set of parallel candidate kernels with identical kernel size $K_{\text{h}}\times K_{\text{w}}$ and input/output channels $C_{\text{in/out}}$. The kernel aggregation is guided by the attention weights and can be expressed as

where $W$ represents the resulting dynamic kernel, $K$ is the number of candidate kernels, $W_{k},k=1,2,\cdots,K$ denotes the $k$-th candidate kernel consisting of $C_{\text{out}}$ filters $W_{k}^{m}\in\mathbb{R}^{K_{\text{h}}\times K_{\text{w}}\times C_{\text{in}}},m=1,2,\cdots,C_{\text{out}}$, and $A_{k}\in\mathbb{R}$ is the attention weight assigned to $W_{k}$. The attention weights are computed using a squeeze-and-excitation mechanism. Specifically, global spatial information is first compressed through global average pooling (GAP), followed by a fully connected (FC) layer, a rectified linear unit (ReLU) activation, another FC layer, and a softmax function. Due to the small size of the convolution kernels and the lightweight nature of the attention mechanism, DyConv is computationally efficient at the cost of an increased number of parameters. Dynamic convolution similarly handles bias, but for simplicity, this detail is not included in the formulas.

The softmax function ensures that the attention weights sum to one, as expressed by $\sum_{k=1}^{K}A_{k}=1$. This normalization helps constrain the kernel space, facilitating the learning of the attention model. During training, candidate kernels with low attention weights are often difficult to optimize. To address this, DyConv incorporates temperature annealing in the softmax function, encouraging a near-uniform attention distribution in the early training stages. Specifically, the features are scaled by a temperature coefficient, which varies during training, before applying the softmax function. A higher temperature produces a flatter distribution of attention weights, facilitating the learning of all kernels. As training progresses, the temperature is gradually reduced, allowing the model to assign more distinct attention weights and improving its ability to select the most relevant kernels.

ODConv [23] introduces a multi-dimensional attention mechanism to enhance the performance of DyConv. It employs a parallel strategy to learn complementary attention weights for convolutional kernels across four dimensions of the kernel space: spatial, channel (input channel), filter (output channel), and candidate kernel dimensions. Specifically, in addition to kernel attention, ODConv assigns distinct attention weights, denoted as $A_{k}^{\text{s}}\in\mathbb{R}^{K_{\text{h}}\times K_{\text{w}}}$, $A_{k}^{\text{c}}\in\mathbb{R}^{C_{\text{in}}}$, and $A_{k}^{\text{f}}\in\mathbb{R}^{C_{\text{out}}}$, to different spatial locations, input channels, and output channels of the $k$-th candidate kernel. It can be represented as

where $*$ denotes element-wise multiplication across different dimensions. The attention weights for each dimension are computed using a multi-head mechanism. The additional spatial, channel, and filter attention weights are all generated using the sigmoid function. For simple implementation, these weights can be shared across all candidate kernels, allowing the subscript $k$ of the corresponding attention descriptor in Eq. 2 to be omitted. By leveraging this refined kernel adjustment mechanism, ODConv achieves comparable or superior performance to standard DyConv while requiring fewer convolutional kernels and significantly fewer extra parameters.

The attention mechanism is pivotal to the effectiveness of adaptive convolution, as it assigns attention weights to each parallel candidate kernel, determining the kernel composition for the current frame. As shown in Fig. 1(b), kernel attention aggregates the input features $Y\in\mathbb{R}^{C\times T\times F}$ along the frequency dimension through power average pooling to derive the descriptor $P\in\mathbb{R}^{C\times T}$ for the time-channel energy distribution, where $C,T,F$ represent the number of channels, frames and frequency bins, respectively. The corresponding pooling is formulated as

where $c$ and $f$ denote the channel and frequency bin indices, respectively. $P$ is subsequently processed through channel modeling and a softmax function to compute the time-varying attention weights $A_{k}(t)$ for each candidate kernel. During training, we omit the temperature annealing strategy described in Section II-A for the softmax function, as our experiments indicate that it does not improve kernel optimization in adaptive convolution. This is likely because different candidate kernels dominate different frames, and the attention weights for many frames do not exhibit one-hot characteristics, eusuring sufficient optimization of all kernels even without temperature annealing.

For causal convolution, an intuitive channel modeling approach is to apply squeeze-and-excitation mechanism to the feature of the current frame. The compressed feature $P$ is processed through an FC layer, a ReLU activation, and another FC layer sequentially. Notably, in the deeper layers of the model, the single-frame channel modeling approach implicitly leverages historical information. This is because preceding layers, such as convolution layers with large temporal receptive fields and inter-frame RNNs, may have already incorporated information from previous frames. Additionally, we explore a multi-frame channel modeling approach, which consists of a stack of a one-dimensional (1-D) convolution layer, a ReLU activation, and an FC layer. The 1-D convolution explicitly incorporates information from previous frames, thereby extending the receptive field of channel modeling. Inspired by TRA [7], a lightweight temporal channel attention mechanism, we adopt a similar strategy by using gated recurrent unit (GRU) to model channel features along the time dimension. Specifically, a GRU followed by an FC layer is employed for temporal channel modeling. This leads to better utilization of inter-frame correlations, thus yielding more efficient adaptive kernels, albeit at the cost of introducing more parameters compared to single-frame modeling.

Benefiting from the lightweight design of kernel attention, adaptive convolution introduces negligible computational complexity to the model, which is critical in practical applications. However, this comes with an increase in parameters, as each adaptive convolution layer contains $K$ parallel candidate convolution kernels, resulting in a convolutional parameter count that is $K$ times the original.

Efficient convolutional blocks, such as residual blocks [17] and ConvNeXt blocks [19], typically consist of multiple depthwise and pointwise convolution layers. Replacing each sub-layer with adaptive convolution results in the natural structure depicted in Fig. 1(c). Given the strong correlations and similarities among hidden features across sub-layers within an individual block, we propose a joint attention mechanism that generates kernel attention weights for multiple sub-layers simultaneously at the beginning of the block using a multi-head strategy. Fig. 1(d) showcases an example of joint multi-layer attention applied to a block with three sub-layers. Specifically, in the channel modeling stage, the final FC layer is replaced with multiple parallel heads, each producing the kernel attention weights for a specific sub-layer. To enhance computational efficiency, we adopt a highly parallelized implementation by increasing the output channels of the FC layer to $N\times K$, where $N$ is the number of sub-layers. The output feature is then split along the channel dimension to derive the kernel attention weights for each sub-layer. The joint multi-layer attention simplifies the model structure by computing the adaptive kernels for all sub-convolutions within a single attention layer. Additionally, it slightly reduces the model size and MACs, particularly in configurations that use temporal channel modeling.

Inspired by ODConv [23], the joint attention mechanism can be further extended to generate attention maps for other dimensions of the kernel space, including the input channel, output channel, and spatial dimensions. They are generated by adding attention heads, followed by sigmoid activation. These optional structures enable more precise adjustment of convolution kernels, enhancing their adaptability to the time-varying speech statistical characteristics.

Adaptive convolution generates independent convolution kernels for each frame in a causal manner, preventing parallel computation across different frames of an utterence. While this limitation does not affect streaming inference, where frames are processed sequentially, it poses a challenge for parallel computing across all frames in an audio sample. In addition to explicitly implementing convolution through matrix multiplication, we propose two approaches to enable multi-frame parallelism.

The first approach involves converting kernel aggregation into the aggregation of convolution outputs. During training, the convolution results for each candidate kernel are computed first, whose results are then aggregated using kernel attention to produce the final output through a weighted sum. This process is mathematically equivalent to adaptive kernel aggregation, as demonstrated by

where $Z(t)$ and $Y(t)$ denote the input and output features of the $t$-th frame, respectively, $Z_{k}$ represents the output obtained by applying the $k$-th candidate kernel independently, which can be computed in parallel as usual, and $\circledast$ denotes the convolution operation. However, this method is restricted to the training stage due to the significantly increase in computational complexity, approximately $K$ times the original.

The second approach leverages grouped convolution to simulate streaming inference while enabling multi-frame parallelism without increasing computational complexity. Inspired by DyConv [21], which addresses parallelism issues for different images in a batch by merging the batch dimension into the channel dimension and assigning each image to a different group, we extend this concept to incorporate both batch and time dimensions. Specifically, an unfold operation is first performed along the time dimension of the input features with kernel size of $K_{t}$, where $K_{t}$ denotes the size of the convolution kernel along the time dimension. This operation extracts a sliding window of features for each time step of the convolution. Subsequently, the batch and time dimensions are merged into the channel dimension, forming an input tensor of shape $\left(1,B\times T\times C,K_{t},F\right)$, where $B$ is the batch size. The reshaped input is then divided into $B\times T$ groups, and convolution is performed with the assembled adaptive kernels.

AdaptCRN works in the short-time Fourier transform (STFT) domain and takes the spectral magnitude as well as the real and imaginary parts as input. Its architecture consists of a spectral compression module, an encoder with 5 layers of adaptive blocks, 2 grouped DPRNN modules, a decoder with 5 layers of adaptive blocks, and a spectral decompression module, as shown in Fig. 2(a). Skip connections are employed between corresponding layers of the encoder and decoder to facilitate optimization and alleviate information loss.

Grouped DPRNN [7] combines grouped RNN [28] and DPRNN [10, 11] to model spectral patterns in a single frame and inter-frame dependence within a certain frequency bin, while reducing the model complexity. Consistent with [7], we adopt grouped GRUs and eliminate representation rearrangement between time steps within the grouped RNN to improve efficiency. Furthermore, in our implementation, we remove the representation rearrangement after the grouped RNN, as the grouped RNN in DPRNN is directly followed by an FC layer, which inherently performs inter-group information fusion. Moreover, representation rearrangement is equivalent to swapping the rows of the FC layer’s weight matrix, which does not alter the final output.

The model predicts the spectral magnitude mask rather than the complex ratio mask (CRM). Our observations indicate that for trained ultra-lightweight models, the imaginary part of CRM is usually close to 0, making it effectively equivalent to a magnitude mask. To enhance mask learning across different frequency bins, we employ a learnable sigmoid function [16, 29] as the activation for the magnitude mask. Further details about the model are described in Secs. IV-B, IV-C, IV-D.

The spectral compression module comprises frequency dimension compression and dynamic range compression. For frequency dimension compression [7], low-frequency components below 2 kHz remain unaltered, while high-frequency bins above 2 kHz are downsampled using a triangular filter based on the equivalent rectangular bandwidth (ERB) scale. This retains crucial harmonic information in low-frequency bins while downsampling the frequency dimension. Dynamic range compression equalizes spectral energy, clarifies the spectral structure, and facilitates the model’s feature extraction. Specifically, logarithmic operation is applied to compress spectral magnitudes, while a magnitude power law is used to compress the real and imaginary parts. The specific spectral compression is formulated as

where $X,X_{\text{r}},X_{\text{i}}$ denote the noisy spectrum and its real and imaginary parts, respectively, the superscript c represents the compressed features, and $f_{\text{ERB}}(\cdot)$ denotes the frequency dimension downsampling operation based on the ERB scale. For the compressed features, we employ the SFE module from [7] to integrate information from adjacent subbands into the channel dimension. The spectral decompression module upsamples the frequency dimension of the single-channel output of the decoder to produce the predicted mask. The decompression is achieved by applying the transpose of the frequency dimension downsampling matrix.

Inverted residual blocks, ConvNeXt blocks, and StarNet blocks are modern CNN designs that have been proven to be lightweight and efficient. These blocks share a similar structure comprising one depthwise convolution (DW Conv) layer and two pointwise convolution (PW Conv) layers. Inspired by these designs, we propose the basic block for AdaptCRN, as depicted in Fig. 2(b). The input features are first processed by layer normalization (LN) along the channel and frequency dimensions, which smooths the energy distribution and preserves spectral patterns across different channels and subbands. The element-wise affine transformation in LN acts as a set of fixed filters to preprocess spectral features. Subsequently, the features pass through a DW Conv layer followed by two PW Conv layers in sequence. Both the DW Conv and the second PW Conv are followed by batch normalization (BN) and parametric ReLU (PReLU) activation, while Gaussian error linear unit (GELU) activation is applied between the two PW Conv layers. We also consider the star operation from StarNet as a replacement for GELU, which is defined as $f_{\text{ReLU6}}\left(x\right)*x$, with $x$ the input features and $f_{\text{ReLU6}}(\cdot)$ the ReLU6 activation function. However, experimental results do not indicate a performance advantage over GELU. Additionally, when frequency dimension downsampling is not required, skip connections are employed to directly add the input features to the output features, mitigating information loss in deeper layers.

By incorporating adaptive convolution into the basic block, we derive the adaptive block, as illustrated in Fig 2(c). This block employs joint attention to generate both the input/output temporal channel attention maps and the adaptive kernel attention weights for the three sub-layers. Notably, adaptive convolution introduces high nonlinearity to the model, which is particularly evident when the activation function between the two PW Conv layers is removed. Typically, two PW Conv layers without activation can be reparameterized into a single layer, expressed as

where $W^{(1)},W^{(2)}$ denote the kernels of the two PW Conv layers, respectively, and the bias term is omitted for simplicity, as its inclusion does not alter the result. However, with adaptive convolution, the reparameterization changes to

making two adaptive PW Conv layers with $K$ candidate kernels equivalent to a single layer with $K^{2}$ kernels. This observation inspires a trade-off between model size and computational complexity. By reducing the number of layers while increasing the number of candidate kernels, we can decrease computational overhead at the expense of increased parameters. Additionally, since the resulting $K^{2}$ kernels and their corresponding kernel attention weights are derived from two groups of $K$ kernels, they reside in a low-rank space relative to directly optimizing $K^{2}$ independent kernels. Consequently, the reparameterized results may exhibit slightly reduced performance compared to directly optimizing a single layer with $K^{2}$ kernels.

We apply the negative scale invariant SNR (SI-SNR) [30] loss and the power-compressed specturm loss as the loss functions, defined as

where $s$ and $\hat{s}$ represent clean and enhanced waveforms, $S$ and $\hat{S}$ are their corresponding spectrograms, the subscripts r, i represent the real and imaginary parts of the spectrograms, respectively, $\langle\cdot,\cdot\rangle$ denotes the inner product operator, and $\text{MSE}(\cdot,\cdot)$ represents the mean squared error (MSE). The overall loss function for model training is given by

where $\lambda_{1},\lambda_{2},\lambda_{3}$ are the empirical weights.

We evaluate our proposed methods on two datasets. The first dataset is constructed using clean and noise samples from the 5th Deep Noise Suppression Challenge (DNS5) dataset [31]. Clean and noise clips are mixed at random SNR levels ranging from -5 dB to 15 dB and scaled to random amplitudes ranging from 0.01 to 0.99 of the full scale to improve the model’s adaptation to magnitude dynamics. Approximately 200 hours of 10-second noisy-clean paired samples are generated for training, while 1,000 samples are generated for validation and testing, respectively. All utterances are resampled to 16 kHz for the experiments.

Additionally, we use the Voicebank+DEMAND dataset [32] to further evaluate the proposed ultra-lightweight model. It is a widely used benchmark for SE tasks, which contains paired clean and pre-mixed noisy speech. The training set comprises 11,572 utterances from 28 speakers, while the test set consists of 872 utterances from 2 speakers. SNR levels for training are set to 0, 5, 10, and 15 dB, and for testing, they are set to 2.5, 7.5, 12.5, and 17.5 dB. All utterances are downsampled from 48 kHz to 16 kHz.

To evaluate the effectiveness of the proposed adaptive convolution, we integrate it into diverse models with varying scales and architectures, including DPCRN [10] of different sizes, DCCRN [9], GTCRN [7], and LiSenNet [16]. DPCRN, as a well-established baseline, serves as a foundational structure for modern CRN-based SE tasks. We utilize three scaled versions of DPCRN, denoted as DPCRN-light, DPCRN-middle, and DPCRN-large.

Ablation experiments are conducted on DPCRN-light to varify the effectiveness of the proposed channel modeling techniques and the joint channel and spatial attention mechanism. Additionally, we validate the advantage of frame-level kernel adjustment by comparing with utterance-level global dynamic convolution, which employs non-causal attention with global pooling. We further investigate the impact of expanding the kernel space by increasing the number of candidate kernels, as well as the performance of directly optimizing the kernels.

For the proposed AdaptCRN, ablation experiments are performed to validate the contributions of incorporating adaptive convolution, joint channel attention, joint multi-layer kernel attention, and dynamic range compression of input features. The nonlinearity introduced by adaptive convolution is validated through experiments on cascaded PW Conv layers and their equivalent single-layer structure. Moreover, by comparing AdaptCRN with DPCRN-light [10], DeepFilterNet [12], GTCRN [7], LiSenNet [16], ULCNet [15], and FSPEN [14], we demonstrate the superior performance of the proposed model.

The STFT is computed with a segment length of 32 ms, an overlap of 50%, a fast Fourier transform (FFT) length of 512, and a square-root Hanning window for analysis and synthesis. For adaptive convolution, the number of candidate kernels $K$ is set to 8, except in experiments specifically investigating its effect. The number of hidden channels for channel modeling is set to 32, while the convolution kernel size of the 1D-Conv used in multi-frame channel modeling is set to 3. Optional joint channel and spatial attention mechanisms are not used unless specified in ablation experiments.

For the three scaled versions of DPCRN, the number of DPRNN blocks is set to 2. The convolution kernel sizes $(K_{t},K_{f})$ for the encoder layers are configured as {(2,5), (2,3), (2,3), (2,3), (2,3)}, with frequency dimension strides of 2 for the first three layers and 1 for the last two layers. The number of output channels per layer is set to {16, 16, 16, 32, 32} for DPCRN-light, {32, 32, 32, 64, 64} for DPCRN-middle and {32, 32, 32, 64, 128} for DPCRN-large. The hidden dimensions of DPRNN are set to 32, 64, and 128, respectively. The decoder mirrors the configuration of the encoder. For GTCRN, LiSenNet, and DCCRN, the model hyperparameter settings remain consistent with their respective original papers, with the exception that the lookahead in the decoder of DCCRN is removed to ensure causality. For GTCRN, its main branch of the GT-Conv block contains three sub-layers and a TRA layer for generating the temporal channel attention map. When the vallina convolution are replaced with adaptive convolution, we employ joint multi-layer kernel attention to generate kernel attention weights for the three sub-layers, and the TRA is replaced with joint output channel attention to produce the temporal channel attention map.

For AdaptCRN, as detailed in Section IV-B, the amplitude, real part, and imaginary part of the noisy spectrum are fed into the spectral compression module. The first 65 low-frequency bands remain unaltered, while the 192 high-frequency bands are mapped to 64 bands. 9-channel, 129-dimensional features per frame are obtained using SFE with a kernel size of 3. The hidden channels of all adaptive blocks are set to 16, except in the final block, where the hidden channels between the two PW Convs are reduced to 4. In the encoder, the convolution kernel sizes of the first two layers are (1,5) with a frequency dimension stride of 2, while the last three layers use kernel sizes of (3,3) with a stride of 1. In the decoder, the last two layers employ depthwise transposed convolutions, with the remaining settings matching the encoder. The grouped DPRNN consists of 2 groups, with a frequency dimension of 33. The hidden channels of the intra-frame and inter-frame RNNs are set to 8 and 16, respectively. The single-channel output of the final adaptive block undergoes spectral decompression followed by a learnable sigmoid function to generate the amplitude mask.

All models are trained using the Adam optimizer with an initial learning rate of 0.001. For DNS5 dataset, 10,000 samples are randomly selected for training in each epoch, with a batch size of 8. For DPCRN-light, GTCRN, LiSenNet, DCCRN, and AdaptCRN, the learning rate is halved at epochs 120, 150, 170, 180, 190, and 200. For DPCRN-middle and DPCRN-large, the learning rate is halved if the validation loss does not decrease for 10 consecutive epochs. For the Voicebank+DEMAND dataset, a cosine annealing scheduler is employed for learning rate decay over 300 epochs with a minimum learning rate of 1e-5. All models are trained with the cost function described in Section IV-D, with $\lambda_{1}=0.01$, $\lambda_{2}=0.7$, and $\lambda_{3}=0.3$.

We employ a set of commonly used speech objective metrics for evaluation, including SI-SNR [30], wide-band perceptual evaluation of speech quality (PESQ) [33], short-time objective intelligibility (STOI) [34] and its extended version (ESTOI) [35], and deep noise suppression mean opinion score (DNSMOS) [36]. DNSMOS is a non-intrusive speech quality metric, which provides three scores including speech quality (SIG), background noise quality (BAK), and overall quality (OVRL). Higher values indicate better performance for all metrics.

A typical audio sample from DNS5 test set is presented in Fig. 3, showcasing noisy and clean spectrograms, enhanced results from DPCRN-light, DPCRN-light integrated with adaptive convolution, and AdaptCRN. This sample poses challenges due to highly non-stationary noise and the presence of two speakers with distinct vocal characteristics, both of which should be preserved in single-channel SE task. The enhanced spectrogram of DPCRN-light exhibits noticeable distortion in one speaker’s voice (highlighted in the green box) and residual noise (highlighted in the blue box), which are effectively mitigated with the incorporation of adaptive convolution. AdaptCRN also achieves superior enhancement quality with ultra-low computational complexity, further highlighting the advantage of the propose methods. Additional audio samples are available at https://github.com/Dahan-Wang/Adaptive-Convolution-for-CNN-based-Speech-Enhancement-Models.

The frame-level kernel attention weights are visualized in Fig. 3(f), demonstrating the relationship between kernel allocation and speech spectral features. The visualization is derived from the third layer of the decoder in DPCRN-light with adaptive convolution, depicting the attention weights of 8 candidate kernels across frames. For clarity, the order of the kernels is manually adjusted during plotting. The results show that the first candidate kernel is predominantly activated during the utterance of the high-pitched speaker, while the segments from the other speaker with a lower pitch are primarily associated with the 2nd to 5th kernels. The 6th to 8th kernels appear to function mainly in pure noise segments. These findings indicate that the adaptive convolution assigns appropriate candidate kernels to each frame based on speech characteristics, effectively facilitating feature extraction and reconstruction.