WaDeNet: Wavelet Decomposition based CNN for Speech Processing

The Naive CNN maps each window of the speech signal directly to its corresponding class. The components of the architecture are as follows:

1. 1.

   Convolutional Block: Each Convolutional Block consists of a Convolutional Layer¹¹1All convolutions refer to one dimensional convolutions unless specified. of kernel size $k$, followed by Batch Normalization and ReLU activation, succeeded by another Convolutional Layer (kernel size = $k$), Batch Normalization and ReLU activation. The first convolution doubles the number of features and maintains the same length of the input feature map while the second convolutional layer maintains the same number of features and divides the length by half. Effectively, for an input feature map of dimension $(F,L)$ the dimension of the output feature map of the block is $(2\times F,L/2)$.

2. 2.

   Fully Connected Block: This block consists of a series of Fully Connected Layers, each being succeeded by ReLU activation and a dropout of probability 0.5 to prevent overfitting. Depending on the complexity of the task, the number of activation units and number of layers is empirically altered.

The Naive CNN comprises of a $N$ Convolutional Blocks followed by a Fully Connected Block. The output feature map of the $n^{th}$ Convolutional Block has dimensions $(2^{n-1}\times c\ ,l/2^{n})$ where $c$ is the number of output channels of the first Convolutional Block and $l$ is the length of the input. The last Convolutional Block’s output feature map is flattened before passing it on to the Fully Connected Block. Softmax activation is applied to the output of the last layer of the Fully Connected Block.

From the analysis of the architecture of the Naive CNN it is clear that there are some shortcomings. Firstly, increasing the depth of this network for better information capture will possibly introduce the problem of vanishing gradients. Secondly, the frequency composition of each window, which is vital for speech processing, is unknown to the model.

In an effort to address these drawbacks, we propose Wavelet Decomposition Net (WaDeNet), an enhancement of the Naive CNN. By inserting an Inception-Residual Block at the end of each Convolutional Block, WaDeNet has an increased network depth while alleviating vanishing gradients. The Inception-Residual Block consists of multiple convolutions with different kernel sizes performed in parallel as shown in Fig. 2. Following this, channel wise concatenation of these representations is done.

The second shortcoming is countered by introducing frequency information into the model by incorporating a multiresolutional representation. This is achieved by embedding the Wavelet Transform into the architecture, which is well experimented with for image reconstruction [11, 12, 13]. As shown in Fig 1, a representation of a particular order of the Discrete Wavelet Transformation (DWT) of the signal is concatenated with the output feature map of the preceeding Convolutional Block ($n$) and passed on to the succeeding Convolutional Block ($n+1$). The Haar wavelet was used for the decomposition of the signal. Each decomposition is passed to a $DWT\ Gate$ which consists of a series of Convolutions that produces the representation. Thus for a total of $N$ Convolutional Blocks there are $N$ levels of wavelet decomposition.

This approach is fundamentally different from using signal processing techniques to extract features that consist of both time and frequency information followed by training a classifier using these features. By incorporating the DWT into the network, the temporal information of the signal is not only retained but also learnt on in an end-to-end manner. Additionally, through feature map concatenation, the number of parameters is drastically reduced while improving performance, as discussed in Section 3.3. This makes it suitable for mobile health applications such has emotion and stress monitoring.

We run our experiments on popular datasets which consist of speech data suitable for mobile health applications such as emotion recognition and pathological voice recognition. The datasets and their respective descriptions are as follows:

1. 1.

   EmoDB [14]: This database consists of 10 actors who simulate 7 different emotions (anger, disgust, fear, happiness, pleasant surprise, sadness, and neutral) by uttering sentences used in everyday communication. There are 535 utterances in the dataset, amounting to 1487 seconds and an average utterance length of about 2.77 seconds.

2. 2.

   RAVDESS [15]: The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) consists of 12 male and 12 female actors speaking each expression in a North American Accent. The actors speak each statement with 8 different emotions - anger, calm, disgust, fear, happiness, sadness, surprise, neutral - at two levels of intensities, except when speaking in a neutral manner.

3. 3.

   VOICED [16]: The VOice ICar fEDerico II Database (VOICED) contains 208 voice samples of adults between the age 18 and 70, of which 150 are pathological and the rest healthy. Each sample is consists of uninterrupted vocalization of the vowel ’a’ for 5 seconds.

4. 4.

   TESS [17]: The Toronto Emotional Speech Set (TESS) consists of 200 words spoken by two female actors. Each of the voice samples emulate 7 different emotions namely anger, disgust, fear, happiness, pleasant surprise, sadness, and neutral.

The voice samples are segmented into windows of 320 ms each with a 75% overlap. Each window is labelled depending on the dataset. For instance, a window in the EmoDB dataset is labelled with the corresponding emotion.

For our experiments, $N=4$, $c_{i}=64$ and $k=3$ for each convolution was chosen. Stochastic Gradient Descent with an initial learning rate of 0.001 was used to optimize the weights of the model by minimizing the categorical cross entropy. WaDeNet was trained for 150 epochs on 60% of the data with a learning rate scheduler which reduced the learning rate by a factor of 10 after 50 epochs. These constants were empirically decided after extensive experimentation.

We compare the performance of the Naive CNN and WadeNet on the datasets discussed in the previous section. Since the F1 score is a more reliable metric for multi-class classification problems, we report it in addition to the overall accuracy of the model’s performance on the test data (20% of the total data).

It is evident that the absence of spectral features translates to a subpar performance as evident from the performance of the Naive CNN (Table 1). In contrast, WaDeNet outperforms the Naive CNN with an average increase in accuracy and F1 score of 13.1% and 13.9% respectively. Additionally, WaDeNet has a maximum improvement up to 38.7% in the F1 score on the RAVDESS dataset.

Furthermore, WaDeNet is considerably lighter than the Naive CNN, with an average reduction of 72.26% in the number of model parameters.

In order to validate WaDeNet, we evaluate WaDeNet’s performance against existing research work that portray state of the art performance on the datasets considered. The experimental conditions for each benchmark were replicated as specified in the respective work. As seen in Table 2, WaDeNet performs considerably better than the current state of the art models across all the datasets, with an average increase in accuracy of 6.36%, with a maximum of 11.4% on the RAVDESS dataset.