Predicting speech intelligibility in older adults for speech enhancement using the
Gammachirp Envelope Similarity Index, GESI ¹¹footnotemark: 1

GESI is designed to objectively predict SI in older adults with age-related HL when listening to sounds enhanced with nonlinear SE algorithms. It is an intrusive measure that requires both the test signal and the original reference signal as inputs. GESI is based on the same auditory model as GEDI [Yamamoto et al., 2020], which was applicable to NH listeners. While GEDI measures the spectral distortion between the reference and test signals, GESI assesses the similarity of auditory representations that reflect HL characteristics. In this study, we extended GESI based on several psychoacoustic observations, such as hearing level, the effect of room acoustics, and temporal processing characteristics. In the following, we first describe the processing flow of the GESI algorithm in section 2.2. A detailed explanation of the signal processing in each block and its rationale is then provided in section 2.3.

Figure 1 shows the block diagram of GESI. The input sounds to GESI are reference ($r$) and test ($t$) signals. First, the cross-correlation between them is computed to perform the time alignment of the speech segment. Then, both signals are analyzed with the gammachirp auditory filterbank (GCFB) [Irino & Patterson, 2006, Irino, 2023], which contains the transfer function between the sound field and the cochlea. The output is the Excitation Pattern (EP) sequence with 0.5 ms frame shift, something like an ``auditory'' spectrogram (hereafter referred to as EPgram). The EP is calculated using the cochlear input-output (IO) function with the absolute threshold (AT) set to 0 dB. The noise floor is represented by a Gaussian noise with a root mean squared (RMS) value of 1 for practical calculations.

The reference speech is always analyzed using the GCFB parameters of a typical NH listener, while the test speech is analyzed using parameters that reflect the individual listener's hearing level, that is either normal or with HL. This allows GESI to reflect cochlear HL resulting from dysfunction of active amplification by outer hair cells (OHCs) and passive transduction by inner hair cells (IHCs). The required input parameters are the hearing level represented by an audiogram, and the compression health parameter ($\alpha$), which indicates the degree of health in the compressive IO function of the cochlea [Irino, 2023]. No dysfunction corresponds to $\alpha=1$ and completely damaged function corresponds to $\alpha=0$. In this study, the hearing level was set to 0 dB and $\alpha=1$ (i.e., NH level) to analyze the reference signal. The individual listener's hearing level and a default value of $\alpha=0.5$, a moderate level, were used to analyze the test signal. This is because the $\alpha$ value cannot be estimated without extensive psychoacoustical experiments. See A for more details.

Next, the time offset between the EPgrams of the reference and test speech is compensated for each channel of the GCFB. The cross-correlation is computed to find the peak position within the maximum correction range of $\pm T_{ma}$, and the time alignment is performed accordingly (see section 2.3.1 for details).

After this correction, the EPgrams are analyzed using an infinite impulse response (IIR) version of the modulation frequency filterbank (MFB) used in GEDI  [Yamamoto et al., 2018, 2020]. The MFB was first introduced in sEPSM [Jørgensen & Dau, 2011, Jørgensen et al., 2013] and implemented using the Fourier transform. However, the IIR version is more phase sensitive, which may lead to better prediction. The upper limit of the modulation frequency of the MFB was limited to 32 Hz as described in section 2.3.2. In addition, the peak gain of each filter in the MFB was set to the corresponding value of the TMTF [Morimoto et al., 2019] of NH and older adults, as described in section 2.3.3.

The internal index is computed using an extended version of the cosine similarity between the MFB outputs for the reference signal ($m_{ij}^{r}(\tau)$) and the test signal ($m_{ij}^{t}(\tau)$):

where $i$ is the GCFB channel $\{i\,|\,1\leq i\leq N\}$, $j$ is the MFB channel $\{j\,|\,1\leq j\leq M\}$, $\tau$ is a time frame number. $\rho$ $\{\rho\,|\,0\leq\rho\leq 1\}$ is a weight value that allows us to handle the level difference between the reference and test sounds. Although $\rho=0.5$ is the original definition of cosine similarity, the level difference is normalized to make the evaluation more difficult. See section 2.3.6 for more details. $w_{i}(\tau)$ is a weighting function applied to each GCFB channel, as described in section 2.3.4.

The overall similarity index $d$ is obtained by weighting and averaging $S_{ij}$ in Eq. 1 by all $i$ and $j$ .

where $w_{j}$ is a weighting function applied to each MFB channel. In this study, $w_{j}=1$ is used, but is adjustable.

The metric $d$ can be converted to word correct score (%) or intelligibility $I$ by the sigmoid function used in STOI [Taal et al., 2011] and ESTOI  [Jensen & Taal, 2016]. That is:

where $a$ and $b$ are parameters determined from a subset of the SI scores in the experimental results using the least squares error method. $I_{max}$ represents the maximum SI specific to the speech dataset used in the experiments. In this study, it was set to 85% based on the SI data of the least familiar words in FW07, i.e., a subset of FW03 [Amano et al., 2009].

Conversion to SI is necessary when calculating the RMS error between the subjective and objective SI. However, this is not necessary to compare the performance of SE algorithms. The metric $d$ can be used directly, as is commonly done in much of the literature (e.g.,  Falk et al., 2015), because the SI score $I$ is monotonically related to the metric $d$ as defined in Eq. 3.

The processing in each block of GESI is explained with extensions from the original version  [Irino et al., 2022, Yamamoto et al., 2023].

The purpose of GESI is to provide information on improvements in SE for older adults with HL. Two important functions are required for such OIMs. First, they must reflect peripheral dysfunctions, including hearing level. Second, they must explicitly represent how the improvement is achieved at the auditory feature level to allow for further improvement. Thus, they must be based on at least a peripheral auditory model. Competitors should have similar functions.

STOI and ESTOI are popular metrics used to evaluate the performance of signal processing. They are simple and effectively used in many SE studies. However, neither metric accounts for individual hearing levels. Recent DNN-based SI models, such as those proposed in the first and second CPC (CPC1 and CPC2) [Barker et al., 2022, 2024], can reflect individual hearing levels. The performance of SI prediction in the competition is excellent. However, it is difficult to understand how the improvements were achieved through signal processing. These models may not provide sufficient information to improve the enhancement process.

HASPI [Kates & Arehart, 2014] and its successors are based on an auditory model and may fulfill the above functions. There are three major versions: the original HASPI [Kates & Arehart, 2014], HASPI version 2 (or HASPIv2)  [Kates & Arehart, 2021], and HASPIw2 [Kates, 2023]. The original HASPI [Kates & Arehart, 2014] is simple and produces several internal metrics that are mapped to an SI score via a sigmoid function. But this function differs from Eq. 3 in that it accepts multiple inputs. Because there are no reported preset values for the parameters, it is difficult to use, and stable prediction values cannot be obtained easily. To address this issue, HASPI has been improved and now exists as HASPIv2 and the latest HASPIw2. Furthermore, previous experiments have shown that the original HASPI has lower predictive performance than the previous version of GESI [Irino et al., 2022], so it was not considered a competitor.

Both HASPIv2 and HASPIw2 produce a single SI score calculated by using NN although there are 10 independent internal metrics. It is possible to derive the SI score for the current experiment by assuming the NN output is considered as a single internal metric $d$ and transformed into by the sigmoid function in Eq. 3. In practice, this method allowed HASPIv2 to serve as the baseline model in CPC2 [Barker et al., 2024]. Although the internal representations of HASPSIv2 and HASPSIw2 are difficult to interpret due to the NN in the output stage, they are preferable to DNN models because they are more compact.

The NNs are trained on English speech samples from the HINT database [Nilsson et al., 1994] and the IEEE sentence [Rothauser, 1969] with different distortions [Kates & Arehart, 2021, Kates, 2023]. HASPIv2 is tuned for the SI of the entire sentence, which includes the higher-level linguistic information. In this experiment, the accuracy rate is based on individual words, so consistency cannot be ensured, making it unsuitable. As the most recent version tuned for keyword SI, HASPIw2 is expected to outperform HASPIv2 in predicting words. Therefore, HASPIw2 was used as the competitor in the SI prediction of the current experiment.

It would also be important to examine the generalizability of HASPIw2 to other languages and experimental conditions to get an idea of the scope of its application.

We calculated SI as the percentage of 4 mora words that the participants answered correctly. Their results are shown as dashed lines in Fig. 3 and 3 along with the predictions of GESI and HASPIw2 described in the next section. There are 15 panels each, with participant IDs assigned from $\rm OA^{\#1}$ to $\rm OA^{\#15}$ in order of participation. There are large individual differences in SI between participants, which may be due to differences in hearing level, temporal characteristics, and cognitive factors. The results of the ``Unpro'' condition (red dashed line) show that the SI score improves with increasing SNR. In contrast, the results of the ``IRM'' condition (blue dashed line) show that the SI score does not change much depending on the SNR. This indicates that the effect of the IRM-based SE is more pronounced at lower SNRs. The SI scores for sounds enhanced by a practical SE algorithm are expected to lie between the two SI curves. These subjective SI scores are the target of the OIM predictions.

The hearing levels of the better ear in Fig. 2 were used for both GESI and HASPIw2. GESI requires compression health ($\alpha$) to specify the slope of the cochlear IO function in GCFB, while HASPIw2 automatically sets the parameter for the similar function. The initial value of $\alpha$ was set to a moderate value of 0.5 for all participants as described in A. GESI can also include the individual TMTF shown in Fig. 2, which has been set to that of the better ear defined in the hearing level.

It is necessary to estimate the sigmoid parameters $a$ and $b$ in Eq. 3 in order to convert the internal metric $d$ to the SI score in this experiment. In Figs. 3 and 3, the subjective SI scores of the ``Unpro'' condition for the first 5 participants ($\rm OA^{\#1}$ to $\rm OA^{\#5}$) were used for this estimation. 20 words were assigned for each participant and each of the 4 SNR conditions (denoted by ``$\rm Unpro^{(closed)}$''). The $d$ values for these 20 words were calculated and averaged to obtain the average metric $\bar{d}$. This $\bar{d}$ was converted to a predicted SI score $\hat{I}$ using the sigmoid in Eq. 3. The values of $a$ and $b$ were estimated by the least-squares method to minimize the error between $\hat{I}$ and the subjective SI score $I$ for the five participants and 4 SNR conditions. SI scores for 20 words were then predicted using these $a$ and $b$ values for all participants, SNR conditions, and signal processing conditions (``Unpro'' and ``IRM''). Therefore, the SI scores in ``$\rm Unpro^{(open)}$'' and ``IRM'' were predicted in the open condition.

The prediction results for the 15 participants are presented as mean and standard deviation for 20 words with error bars in Figs. 3 for GESI and 3 for HASPIw2. The ``$\rm Unpro^{(closed)}$'' shows the conditions under which the ``Unpro'' SI scores were used to determine the parameters $a$ and $b$ of the sigmoid function in Eq. 3. The ``$\rm Unpro^{(open)}$'' shows the case where the prediction was performed with the open condition using the $a$ and $b$ derived above. Note that the prediction for the ``IRM'' sound was always evaluated with the open condition.

GESI generally predicted quite well for all participants for both ``Unpro'' and ``IRM''. The major differences are only noticeable in the cases of $\rm OA^{\#8}$ and $\rm OA^{\#9}$ for underestimation and $\rm OA^{\#10}$ for overestimation. In contrast, the predictions of HASPIw2 were not as good as those of GESI. In particular, the standard deviation of 20 words is much larger than that of GESI. The average SI scores for $\rm OA^{\#4}$, $\rm OA^{\#5}$, $\rm OA^{\#6}$, $\rm OA^{\#8}$, and $\rm OA^{\#9}$ are much more overestimated than those in GESI. In the case of $\rm OA^{\#5}$, it should have made better predictions even for HASPIw2 because the subjective SI score was used to determine the sigmoid parameters $a$ and $b$ as in ``$\rm Unpro^{(closed)}$'', but it did not. This is probably because the metric value estimated by HASPIw2 is highly variable across words and its mean is always well above zero. As described in section 2.4, HASPIw2 uses the NN trained by English keywords. The current results suggest that the generalization ability to this experiment using Japanese words is not very high.

The above results were calculated using an efficiency coefficient $\eta$ of 0.7, which was determined in a preliminary study. The validity of this value was not demonstrated. Therefore, we investigated the extent to which $\eta$ affects the error. Figure 4 shows the RMS error of individual words when $\eta$ is varied from 0 to 1 in increments of 0.1. The open and closed conditions were the same as in the above prediction shown in Figure 3.

The curves changed gradually, with no unusual increases or decreases that would suggest overfitting. In the ``Unpro'' condition, the minimum value was achieved at $\eta=0.8$, while in the ``IRM'' condition, the minimum value was achieved at $\eta=0.7$. Since ``Unpro'' includes both closed and open conditions, whereas ``IRM'' only includes open conditions, its overall RMS error value is lower than that of ``IRM''. The standard deviation is approximately 6% at $\eta=0.7$ or $0.8$, which is lower than for the other $\eta$ values. This indicates reduced variability in predictions. Therefore, a value of 0.7 seems to be reasonable for the current prediction.

In this way, GESI's interpretable parameter $\eta$ can be determined through learning using data from a large number of listeners. However, it is thought that adjusting the value of $\eta$ for individual listeners is more effective. This value is thought to allow for the simple introduction of factors that cannot be expressed by the bottom-up model of GESI, such as the influence of the mental lexicon, listening effort, and cognitive factors, into a single value. However, this is beyond the scope of this paper, and further research is awaited.

Statistical analysis was performed to confirm whether the SI prediction described above is stable and generalizable across different combinations of open and closed sets in GESI and HASPIw2.

For this purpose, different 5 participants were randomly selected to determine the sigmoid parameters $a$ and $b$ from their SI scores of the ``Unpro'' condition, and all SI scores were predicted as described above. This prediction was repeated 9 times for a total of 10 results. Furthermore, this procedure was also performed when the TMTF was not introduced in GESI, by substituting the HL parameters for the NH parameters in Eq. 5, i.e. $A_{j}^{t}=A_{j}^{r}$. Then the RMS errors were computed in two ways described below and accumulated for three processing conditions of ``$\rm Unpro^{(closed)}$'', ``$\rm Unpro^{(open)}$'' and ``IRM''.

GESI was developed to evaluate the effectiveness of signal processing techniques, such as SE and noise suppression, for older individuals with HL. GESI is a bottom-up model configured with only a few explicitly defined parameters, in order to clarify the improved internal representation by signal processing. The values of the sigmoid parameters $a$ and $b$ in Eq. 3 depend on the evaluation material and conditions. They require a few matched data sets of signal and subjective SI scores. However, these two parameters are unnecessary when GESI is used for performance comparisons.

This feature also highlights a fundamental limitation of GESI. The current version of GESI does not model the factors involved in central auditory and cognitive processes. Therefore, it is impossible to differentiate the SI scores of listeners with similar hearing level profiles but different central and cognitive dysfunctions or improvements. Furthermore, only two parameters, $a$ and $b$, connect the bottom-up metric and the SI score. These are insufficient for incorporating all the complex, higher-order information that depends on listeners and situations. This differs from recent DNN-based models, which can make more accurate predictions with a large amount of training data.

Nevertheless, when parameters $a$ and $b$ are determined using a part of individual's SI scores, it is possible to fairly evaluate the effectiveness of signal processing for that individual, as demonstrated under the IRM conditions shown in Fig. 3. This may provide valuable information to improve signal processing. This is an important application of GESI.

GESI has another usage. Depending on the hearing level setting, GESI can predict SI scores for both older adults with HL and young NH individuals. Additionally, GESI only models bottom-up processes. These properties could be used to distinguish between peripheral and central factors contributing to the SI score [Yamamoto et al., 2025]. First, the parameters $a$ and $b$ in the sigmoid are estimated using the SI scores of multiple young NHs who are expected to have normal cognitive function. Next, the hearing level of a specific older adult is set and their SI scores are predicted based on these values of $a$ and $b$. If the predicted SI score is worse than the score obtained in the subjective experiment, it can be estimated that this older adult has dysfunction in their higher-level processing or cognitive factors. Conversely, a higher predicted SI could suggest greater efficiency in using the mental lexicon or in other higher-level processing in response to gradually declining peripheral hearing level. This method could help to identify the cause of HL.

In this study, we tried to incorporate the TMTF of individual listeners into GESI. However, as shown in 5, there was no difference between the GESI results with and without the TMTF. There are two possible reasons for this.

First, the TMTFs of the older participants were not as poor as those of the neuropathy patients, whose $N_{ps}$ were greater than -10 dB on average [Narne, 2013]. As a result, the effect of the TMTF may not have been large enough to be observed. However, it remains difficult to conclude whether this is a reasonable explanation.

The second reason is a technical issue that should be improved. The TMTF introduction method described in section 2.3.3 was imprecise. The TMTF measurement described in B was performed with broadband noise to simplify and expedite the process. Then, the estimated low-pass characteristics were uniformly reflected in the peak gain of the MFB, regardless of the frequencies in the GCFB channels. This may have negatively impacted the results. The temporal characteristics may differ from channel to channel. Furthermore, modulation differences can only be detected at low frequencies within the audible range for older adults with age-related HL. Therefore, the measured results may not reflect the temporal characteristics of high frequencies. To avoid this issue, the TMTFs should be measured at each audiometric frequency. If a sinusoid is used for simplicity, the response curve will not become a simple low-pass filter because of the sideband components produced by the modulation [Kohlrausch et al., 2000]. However, this does not affect the current version of GESI since the minimum modulation depth remains nearly constant below 32 Hz. Alternatively, bandpass noise could also be used for the measurement.

The method of introducing the TMTF into the MFB should also be considered more carefully. In this study, the measured TMTF was applied directly to the peak gain of the MFB. However, compensation may be necessary because the auditory filter has compressive characteristics. It is important to take these issues seriously and incorporate them into the design of OIMs.

The results shown in section 4 suggest that GESI is more advantageous than HASPIw2 in predicting current experimental results. Therefore, HASPIw2 is not sufficiently capable of generalizing to Japanese word SI prediction. It may be possible to train the NNs in HASPIw2 using compatible Japanese word data to improve performance. However, with such a strategy, the training would likely need to be tailored for each language, experimental condition, and listening condition.

Conversely, we also examined whether GESI could generalize to English SI prediction. Here, we compared HASPIv2 and GESI using CPC2 data [Barker et al., 2024]. The rationale for using CPC2 data and the comparison with HASPIv2, the evaluation method, and the results are presented in detail in D. It was difficult to conduct a comparison as rigorous as the prediction presented here because of missing information of the SPL in the CPC2 data. However, the evaluation was designed to be fair by aligning the conditions as much as possible.

The results of D show that the RMS errors in the open test were not significantly different between HASPIv2 and GESI. In theory, GESI is not superior to HASPIv2 for the CPC2 task because GESI cannot represent higher-order linguistic knowledge, whereas HASPIv2 can. Nevertheless, similar performance was achieved, and GESI is reasonably applicable for predicting English SI with word segmentation.

It is also worth considering the complexity of the models by examining the necessary parameters. Both models use the sigmoid parameters, $a$ and $b$. HASPIv2 additionally uses NNs with ten input units (plus one constant unit), four hidden units (plus one constant unit), one output unit. This configuration yields a total of more than 44 uninterpretable weight parameters. In contrast, GESI uses only two explicitly defined parameters $\rho$ and $\eta$. This suggests that the same level of performance could be achieved with far fewer parameters. Following Occam's razor principle [McFadden, 2021], GESI is clearly simpler and more efficient. Furthermore, this result provides valuable insight into how peripheral processing affects SI.