Robust classification using hidden Markov models and mixtures of normalizing flows

Phonemes can be thought of as one of the fundamental units of speech sounds, that indicate the pronunciation of a word. A spoken utterance, which is essentially a sequence of words, is composed of phonemes. The task of phone recognition involves creating a system that can be given input as a human spoken utterance, and the system outputs a sequence of phones that indicate the pronunciation of the actual spoken text. Speech phone recognition is popular in the research community because it is a fundamental task in developing a speech recognition system [8]. It is also inherently free from the limitations of vocabulary. The phone recognition accuracy continues to improve [9].

Neural networks are recently much used for speech recognition, mainly in discriminative training setups. A list of neural network based methods and their performances for phone recognition can be found in the Github site [9]. An earlier attempt is to use restricted Boltzmann machines (RBMs) to form a deep belief network (DBN) that served as the acoustic model for HMM [10]. The DBN used a softmax output layer and was trained discriminatively using backpropagation, achieving a phone recognition accuracy of around 77% on the TIMIT dataset.

Dynamical system models have also been used for sequence-to-sequence classification. The dynamical neural networks are recurrent neural networks (RNNs), long-short-term-memory networks (LSTMs), and their gated recurrent unit based simplifications. Attention mechanisms can further improve performance. Example works are [11], [12]. Almost all these example works use discriminative training. All these methods used a softmax function to represent the phone probability estimation. The best phone recognition accuracy result so far known using discriminative training is $85.1\%$ [13]. Finally, the authors in [14] proposed a hybrid HMM based on PyTorch-Kaldi toolkit, whose phone recognition accuracy is mentioned as $86.2\%$.

The framework used for implementing the normalizing flow-based model is a Hidden Markov Model (denoted by $\mathbf{H}$). An HMM is basically composed of the following quantities: the set of hidden states of the HMM denoted by $\mathrm{S}$, the initial probability vector for the states of the HMM denoted by $\mathbf{q}$, the state-transition probability matrix denoted by $\mathbf{A}$, and the output probability distribution for each state $s$ denoted by $p\left(\mathbf{x}|s;\mathbf{\Psi}_{s}\right)$ and parameterised by a set of parameters $\mathbf{\Psi}_{s}$.The probabilistic model of the NMM-HMM for each hidden state is a weighted mixture of $K$ density functions that is defined as:

In (2), the weights $\pi_{s,k}$ denote the probability of drawing a given component $k$ from a categorical distribution with $\pi_{s,k}=p\left(k|s;\mathbf{H}\right)$ and they satisfy $\sum_{k=1}^{K}\pi_{s,k}=1$ for each s. The feature vector $\mathbf{x}$ is considered to be generated from a flow-based generator function $\mathbf{g}_{s,k}:\mathbb{R}^{D}\to\mathbb{R}^{D}$, such that $\mathbf{x}=\mathbf{g}_{s,k}\left(\mathbf{z}\right)$, where $\mathbf{z}$ is a D-dimensional latent variable that is drawn from a known prior distribution. As we are considering normalizing flow models, $p\left(\mathbf{z}\right)$ is assumed to be a standard multivariate normal distribution. Assuming the function $\mathbf{g}_{s,k}$ is invertible, and the corresponding normalizing function is $\mathbf{f}_{s,k}$ (or equivalently $\mathbf{g}^{-1}_{s,k}$), s.t. $\mathbf{z}=\mathbf{f}_{s,k}\left(\mathbf{x}\right)$, we have:

This equation shows that the density computation is exact and tractable. In practice, a logarithm is applied on both sides of (3) to transform it into a sum, and the log-likelihood of the signal is calculated using the log-probability of the prior and the log-determinant of the Jacobian.

The NMM-HMM model is intended to be used for modeling a sequential signal such as $\mathbf{\underaccent{\bar}{x}}$, having an empirical distribution of the dataset as $\hat{d}\left(\mathbf{\underaccent{\bar}{x}}\right)$ and the maximum likelihood maximization problem that is required to solve is:

where $r$ denotes the index of the sequential signal in the training data, $R$ denotes the total number of sequential input signals under consideration and $H$ denotes the HMM model under consideration from the possible hypothesis set of models. This problem can be solved using the well known Expectation-Maximization framework. The Expectation step (”E-step”) involves the calculation of the posterior probability distribution of hidden sequences $\mathbf{\underaccent{\bar}{s}}$ and $\mathbf{\underaccent{\bar}{k}}$ to obtain an expected value of the likelihood of the data sequence $\mathbf{\underaccent{\bar}{x}}$ under the current model parameters $\mathbf{{H}^{old}}$.”. This is shown as follows:

The second step consists of the Maximisation step (”M-step”) consists of finding the model $\mathbf{H}$ that maximizes the expected log-likelihood computed in (5). This can be decomposed into three separate maximization problems as in (6):

where,

The maximisation problems in (7), (8) can be solved using standard EM forward-backward algorithm to compute the posterior distribution effectively, explained in [15]. For solving the maximisation problem of the last equation 9, we need to maximise the likelihood with respect to the mixture of weights and the set of parameters of the flow model. This would require the computation of the log-determinant for the flow model that is explained in the next section.

Each generator function is realised as a flow model. It maps a given observation from the feature space to a latent space of the same dimension. Every mapping from layer to layer should be bijective (i.e. one-to-one and invertible) and log-determinant in the inverse direction should be easy to compute. The signal flow for a flow model having $L$ layers can be illustrated as follows:

where $\mathbf{f}_{s,k}^{[l]}$ denotes the $l^{th}$ layer network of $\mathbf{f}_{s,k}$, and each such $\mathbf{f}_{s,k}^{[l]}$ is invertible. Flow models have been first proposed in [5]. Different flow model architectures may have different kinds of coupling between two successive layers in the network or some other bijective mapping. To illustrate a small section of the mapping from the data space to the latent space, let us consider the input feature at the $l^{th}$ layer denoted by $\mathbf{h}_{l}$. This feature is mapped to $\mathbf{h}_{l-1}$ using the function $\mathbf{f}_{s,k}$. At every layer the $D$-dimensional input feature is split into two parts. Let us assume the features are $\left[\mathbf{h}_{l,1:d},\mathbf{h}_{l,d+1:D}\right]^{T}$ (where $d$ denotes the number of components in the first sub part). The relation is as follows:

The inverse function $(\mathbf{f}:X\to Z)$ (from data space to latent space, which is used in Jacobian computation during training) is defined as:

(where $\odot$ denotes element-wise multiplications, $\mathbf{s}:\mathbb{R}^{d}\to\mathbb{R}^{D-d}$, $\mathbf{t}:\mathbb{R}^{d}\to\mathbb{R}^{D-d}$, with $\mathbf{s,t}$ being shallow feed-forward Neural Nets that differ only in the activation function for the last layer, which is a hyperbolic tangent (tanh) activation function for the scale function $\left(\mathbf{s}\right)$ (modeling logarithm of standard deviation) and an identity activation for the translation function $\left(\mathbf{t}\right)$[16]. For the flow model, the inverse mapping shown in (12) is computed for every layer and the determinant of the Jacobian matrix is computed as the product of the determinants of the Jacobian matrices at every layer:

where each $\det\left(\nabla\mathbf{f}_{s,k}^{[l]}\right)$ is computed as:

In (14), $\mathbf{I}_{1:d}$ denotes an identity matrix and $diag(\dotsc)$ denotes a diagonal matrix with the elements of the vector in the main diagonal. It should be kept in mind and an alternate ordering between the two parts of the signal $\left[\mathbf{h}_{l,1:d},\mathbf{h}_{l,d+1:D}\right]^{T}$, described in the affine coupling layer in (11), is required to avoid the problem of partial identity mapping [16] in the model. For solution of (9), the problem can be broken down into two parts: learning the set of mixture of weights: $\mathbf{\Pi}=\{\mathbf{\pi_{s}}|s\in S\}$, and the learning the set of flow model parameters: $\mathbf{\Phi}=\{\mathbf{\phi_{s}}|s\in S\}$. This is shown as:

The problem of learning the mixture of weights can be solved using a simple lagrangian formulation while problem of learning the flow model parameters can be solved by using the results of the change of variable (3) and the log-determinant derived in (14) as:

The first term on the right hand side of (16) is basically an expectation computed over the log-probability of latent data derived using the normalizing function $\mathbf{f}_{s,k}:X\to Z$, and the second term is the result of the log-determinant of the Jacobian that is computed using (13) and (14). The essential steps of the learning procedure are described in psuedocode in Algorithm 1.

The experiments were carried out using the TIMIT dataset [8], which has utterances labelled at the phoneme level. The speech signal in the dataset has been sampled at 16 kHz, and the dataset consists of 6300 phoneme-level speech utterances that have been split into two sets - A training set consisting of 4620 utterances and a testing set consisting of 1680 utterances. In the original TIMIT dataset there are 61 phones available for classification. There also exists a smaller, ’folded’ set of 39 phones mapped from the larger set of phones [8]. The experiments were performed for classification on the folded set of 39 phones. Four varieties of additive noises from from the NOISEX-92 database (also sampled at the same frequency of 16 kHz) were considered and snippets were taken from random offsets in the noise database and were added to the clean test data at different Signal to Noise ratio levels (with respect to the clean test signal) to generate noisy test sets. The features used were Mel frequency Cepstral Coefficients (MFCCs) that were extracted on a per frame basis (25 ms window frames, with a 10ms time shift between successive windows). This resulted in a vector of 13 MFCCs, along with column-wise concatenation of velocity ($\Delta$ coefficients) and acceleration coefficients ($\Delta\Delta$ coefficients) into a 39-dimensional feature vector. These feature vectors were subsequently processed, split into training and test sets and arranged on a per-class basis i.e. for each of the 39 phonemes present. This helped to set up the conditions for generative training of the models.

The NMM-HMM models for each of the 39 classes of phones were trained using generative training. Training was carried out using Adam [17] optimization, with data processed on a batch-wise basis. For each HMM model, the number of states used was clipped between 3 and 5, with the exact number of states used depending upon the mean sequence length of the incoming signal. For the GMM-HMM model, diagonal covariance matrices were considered for modeling each component Gaussian in the model, and the no. of mixture components were to be decided based on the basis of model performance. The state-transition matrix $\mathbf{A}$ for each NMM-HMM model was initialised as an upper triangular matrix. For each NMM-HMM model, a flow block was used to refer to a pair of consecutive coupling layers that have been described in (11),(12). It was necessary to ensure that the signal in the $l^{th}$ layer would be alternated by the ${(l+1)}^{th}$ layer, so that there is a mixing of the signals between consecutive coupling layers and no identity mapping occurs. Each NMM-HMM model consisted of four such flow blocks in the implementation, which was found to work well. The evaluation was done using a full forward-backward algorithm, with the metric used as accuracy $(100-PER\%)$ computed on the test set, as a percentage of the number of correct predicted phonemes among the total set of phonemes.