Parametric Object Coding in IVAS: Efficient Coding of Multiple Audio Objects at Low Bit Rates

The mono input objects $x_{i}$ are transformed into a time-frequency representation $X_{i}(k,n)$ by means of a filterbank, where $k$ and $n$ denote frequency bin and time slot indices, respectively. In IVAS, we use a Modified Discrete Fourier Transform analysis filterbank as defined in [2] which converts $20$ \unitms frames into $4$ time slots and $240$ frequency bins at a sampling rate of $48$ \unitkHz. For each time-frequency tile $(k,n)$, the signal power of the $i$-th object is then determined as $P_{i}(k,n)=|X_{i}(k,n)|^{2}$. To reduce the data load, the $4$ time slots are grouped into one slot and the $240$ bins are grouped into $11$ psychoacoustically designed bands. The parameter band borders are given by $B(l)$ and are specific to the input sampling rate. The signal power values per parameter band $l$ are then summed according to:

For each parameter band, the two objects with the greatest power values are identified and a power ratio between them is determined:

Note that $P_{1}(l)$ denotes the power of the most dominant object within the parameter band and $P_{1}(l)\geq P_{2}(l)$. As both ratios sum to one, $r_{2}(l)$ is obtained by $r_{2}(l)=1-r_{1}(l)$ and does not need to be transmitted. $r_{1}(l)$ is quantized with three bits using:

With $L=11$ parameter bands, the side information bit expenditure for each frame of $20$ \unitms amounts to $4L$ bits for the dominant object identifications–each value is expressed with $2$ bits–and $3L$ bits for the power ratio indices. Once per frame, the quantized azimuth ($7$ bits) and elevation ($6$ bits) information of each object is also transmitted.

The input audio objects $x_{i}$ are downmixed into two downmix channels using downmix gains that are derived from two cardioids and depend on the object positions–more specifically, the azimuth information $\theta_{i}$ of each object. The two cardioids are fixed and oriented towards 90° (left) and -90° (right), respectively. The downmix gains $w_{L,i}$ and $w_{R,i}$ for the left and right channel are given by:

The downmix channels are then obtained by:

By taking into account the positions of the objects, all objects located in the left hemisphere are assigned a stronger weight for the left channel, while objects located in the right hemisphere are assigned a stronger weight for the right channel. This facilitates the rendering process. Our experiments further showed that one downmix channel is not enough to produce satisfactory results.

To prevent discontinuities, smoothing is applied between frames, and an energy compensation is conducted to account for differences between the power of the two dominant objects and the total power of all objects. The left and right downmix channels are encoded independently and transmitted to the decoder. Details on the core coding aspects are outside the scope of this paper and can be found in [2].

The power ratios are obtained from the transmitted power ratio index:

Together with the decoded direction information and the object indices of the two most dominant objects per parameter band, the output signals can be created from the two decoded downmix channels. For this, the downmix is first transformed into a time-frequency representation by means of a complex low-delay filterbank [18] which converts frames of $20$ \unitms into $16$ time slots and $60$ frequency bins. The analysis filterbank at the decoder differs from the analysis filterbank at the encoder to allow for a consistent rendering across different IVAS modes.

IVAS supports, among other output formats, the four loudspeaker layouts 5.1, 5.1.4, 7.1, and 7.1.4 according to the CICP setups as specified in [19]. To render the downmix to a specific loudspeaker layout, covariance synthesis [20] is employed where a mixing matrix $\mathbf{M}$ is computed per time-frequency tile which transforms the transmitted downmix signals into the loudspeaker output signals

where $\bm{x}=[X_{1}(k,n),X_{2}(k,n)]^{T}$ denotes the decoded downmix signals, $\bm{y}=[Y_{1}(k,n),Y_{2}(k,n),\ldots,Y_{S}(k,n)]^{T}$ denotes the loudspeaker signals, and $S$ is the number of loudspeakers.

In order to compute the mixing mixing matrix $\mathbf{M}$, the covariance synthesis requires an input covariance matrix $\mathbf{C}_{x}$, a target covariance matrix $\mathbf{C}_{y}$, and a prototype matrix $Q$. The prototype matrix $Q$ is dependent on the configured loudspeaker layout and is computed by mapping $DMX_{L}$ to all left-hemisphere loudspeakers and $DMX_{R}$ to the right-hemisphere loudspeakers. The center speaker is assigned both channels with a weight of $0.5$.

Since the input objects are uncorrelated, the input covariance matrix $\mathbf{C}_{x}$ is computed by considering only the main diagonal elements. It simplifies to:

In order to compute the target covariance matrix $\mathbf{C}_{y}$, we need panning gains (also referred to as direct responses $dr$), a reference power $P_{DMX}$, and the decoded power ratios $\hat{r}_{i}$ which were calculated per parameter band $b$ and are now valid for all frequency bins contained in $b$. The target covariance matrix $\mathbf{C}_{y}$ is given by

where $\mathbf{R}=[\bm{dr}_{1},\bm{dr}_{2}]$ denotes the panning gains corresponding to the two dominant objects and $\mathbf{E}$ is a diagonal matrix with element $e_{i,i}=E_{i}=DP_{i}(k)$ containing the power corresponding to the $i$-th dominant object. It is given by

where $P_{DMX}(k)=\sum_{i=1}^{2}\sum_{n=0}^{15}|X_{i}(k,n)|^{2}$. The panning gains are computed for the two dominant objects using Edge Fading Amplitude Panning (EFAP) [21], an improved panning method compared to Vector-Base Amplitude Panning (VBAP) [22], based on their corresponding direction information and the target loudspeaker layout. A synthesis filterbank is applied on the output signals $\bm{y}$ to obtain the time-domain loudspeaker signals for playback.

A set of twelve audio scenes containing different kinds of audio objects is used for the subjective evaluation of the proposed method. Since multi-party conferencing is probably the most relevant use case, most of the audio scenes feature speech uttered by different participants. The amount of overlapped speech varies from scene to scene, but mostly, it is kept small to create realistic scenarios of polite discussions. IVAS also supports music and mixed audio, which is why some audio scenes contain general audio content in the form of music (different instruments and vocals) and ambient sound.

Table I summarizes the details of each audio scene of the test set. Azimuth and elevation values are given for each of the $4$ objects of the scene; in case of a present moving object, the start and end position is given and the motion itself is uniform across all frames of the audio file. The duration of the test files varies between $10$ and $21$ seconds at $48$ \unitkHz.

With the test set, two subjective listening tests following the Multiple Stimuli with Hidden Reference and Anchor (MUSHRA) methodology [23] were conducted on a 7.1.4 loudspeaker layout. The first test focuses on audio scenes containing three objects, while the second test focuses on scenes with four objects. In the three-object test, the fourth objects of the scenes are not used. The test comprises the following conditions: Hidden reference, 3.5 \unitkHz low-pass anchor, ParamISM at 24.4 \unitkbit/s, multi-mono EVS at 9.6 \unitkbit/s per object, and multi-mono EVS at 8 \unitkbit/s per object. For the four-object test, ParamISM at 32 \unitkbit/s is used instead of 24 \unitkbit/s. The rest of the conditions remain the same.

For multi-mono EVS, the coded objects were rendered to the loudspeaker layout using the unquantized object metadata in a post-processing step. The gains for the rendering were computed using EFAP [21]. In this context, it is important to note that the metadata bit rate is part of the total bit rate in IVAS. In contrast, EVS uses the full bit rate for coding the objects without taking into account any metadata bit rate. This puts multi-mono EVS at an advantage over IVAS when the exact same bit rates are compared. On average, the metadata consumes $1.6$ \unitkbit/s per object in IVAS.

Figure 2 shows the results of the listening tests. A total of $13$ expert listeners participated in both tests. The results show that ParamISM in IVAS achieves an audio quality in the good range for speech and in the fair range for music and other audio content. On average, it outperforms EVS at the same bit rate by $15$ to $20$ MUSHRA points. Notably, IVAS ParamISM supports super-wideband coding at the investigated bit rates, whereas multi-mono EVS at $8$ \unitkbit/s per object only supports wideband coding. At $9.6$ \unitkbit/s per object, EVS also supports super-wideband coding. Especially in the four-object test, it is seen that ParamISM approaches the audio quality of EVS at the higher bit rate, while also including the coded metadata in the total rate. Subtracting the metadata bit rate, ParamISM indeed uses only $61$% of the rate of $4$x EVS at $9.6$ \unitkbit/s for coding the audio objects. In the three-object case, ParamISM uses $63$% of the rate of $3$x EVS at $9.6$ \unitkbit/s.

The quality of ParamISM drops somewhat in audio scenes that contain strong object overlaps, e.g., audio scenes i1 and i5 where there is a 50-100% overlap of the talkers. One can argue, though, that such an overlap would not occur in a realistic conferencing or conversation scenario. Scene i11 contains a recording of a choir, where, in addition to a 100% temporal overlap, there is also the challenge of a large overlap in the frequency components. Scene i6 proves somewhat challenging due to simultaneously occurring transients in all objects. For pure music content, a higher bit rate is required to achieve good to excellent quality and for this, IVAS provides multiple other modes in the form of multi-channel and Ambisonics coding or non-parametric object coding at bit rates of up to $512$ \unitkbit/s. However, ParamISM is still able to produce acceptable quality at low bit rates, retaining immersion for any spatial audio scene, while at the same time fulfilling the low-delay and complexity requirements of a communication codec.

To also evaluate ParamISM in an objective manner, Table II summarizes the complexity measured [16] in WMOPS for both IVAS ParamISM and multi-mono EVS. Measurements are conducted for ParamISM with three and four input objects. Accordingly, three and four instances of EVS are run and measured for comparison. The numbers show that the parametric object coding mode in IVAS generally requires fewer WMOPS to encode and decode audio scenes with three or four objects than multi-mono EVS.