Deep ad-hoc beamforming

Deep learning based speech enhancement has demonstrated its strong denoising ability in adverse acoustic environments [Wang & Chen, 2018], which has attracted much attention since its first appearance [Wang & Wang, 2013]. Although many positive results have been observed, existing deep-learning-based speech enhancement and its applications were studied mostly with a single microphone or a conventional microphone array, such as a linear array in a portable equipment. Its performance drops when the distance between the speech source and the microphone (array) is enlarged. Finally, how to maintain the enhanced speech at the same high quality throughout an interested physical space becomes a new problem.

Ad-hoc microphone arrays provide a potential solution to the above problem. As illustrated in Fig. 1, an ad-hoc microphone array is a set of randomly distributed microphones. The microphones collaborate with each other. Compared to the conventional microphone arrays, an ad-hoc microphone array has the following two advantages. First, it has a chance to enhance a speaker’s voice with equally good quality in a range where the array covers. Second, its performance is not limited to the physical size of application devices, e.g. cell-phones, gooseneck microphones, or smart speaker boxes. Ad-hoc microphone arrays also have a chance to be widespread in real-world environments, such as meeting rooms, smart homes, and smart cities. The research on ad-hoc microphone arrays is an emerging direction [Markovich-Golan et al., 2012, Heusdens et al., 2012, Zeng & Hendriks, 2014, Wang et al., 2015, O’Connor et al., 2016, O’Connor & Kleijn, 2014, Tavakoli et al., 2016, Jayaprakasam et al., 2017, Tavakoli et al., 2017, Zhang et al., 2018, Wang & Cavallaro, 2018, Koutrouvelis et al., 2018]. It contains at least the following three fundamental problems:

• 1.

  Channel selection. Because the microphones may distribute in a large area, taking all microphones into consideration may not be the best way, since that the microphones that are far away from the speech source may be too noisy. Channel selection aims to group a handful microphones around a speaker into a local microphone array from a large number of randomly distributed microphones.

• 2.

  Device synchronization. Because the microphones are distributed in different positions and maybe also different devices, their output signals may have different time delay, clock rates, or adaptive gain controllers. Device synchronization aims to synchronize the signals from the microphones, so as to fascinate its following specific applications.

• 3.

  Task-driven multichannel signal processing. It aims to maximize the performance of a specific task by, e.g., adapting a multichannel signal processing algorithm designed for a conventional array to an ad-hoc array. The tasks include speech enhancement, multi-talker speech separation, speech recognition, speaker recognition, etc.

However, current research on ad-hoc microphone arrays is still at the beginning. For example, some work discussed a so called message passing problem between microphones where it assumes that the ground-truth noise spectrum is known [Heusdens et al., 2012] or the steering vectors from speech sources to microphones are known [Zeng & Hendriks, 2014]. Some work focused on the channel selection problem in an ideal scenario where perfect noise estimation and voice activity detection are available [Zhang et al., 2018]. Although some work tried to jointly conduct noise estimation and channel selection by advanced mathematical formulations, it has to make many assumptions such as the ground-truth distances between microphones, ground-truth geometry of the array, and free-space signal transmission model without reverberation [O’Connor et al., 2016].

A possible explanation for the above difficulty is that an ad-hoc microphone array lacks so much important prior knowledge while contains so many interferences that we finally have little information about the array except the received signals in the extreme case. To overcome the difficulty, we may consider the way of learning the priors, parameters, or hidden variables of the array instead of making unrealistic assumptions. Supervised deep learning, which learns prior knowledge and parameters by neural networks, provides us this opportunity as it did for the supervised speech separation with the conventional microphone arrays [Wang & Chen, 2018].

In this paper, we propose a framework named deep ad-hoc beamforming (DAB) which brings deep learning to ad-hoc microphone arrays. It has the following four novelties:

• 1.

  A supervised channel selection framework is proposed. It first predicts the quality of the received speech signal of each channel by a deep neural network. Then, it groups the microphones that have high speech quality and strong cross-channel signal correlation into a local microphone array. Several channel selection algorithms have been developed, including a one-best channel selection method and several N-best channel selection methods with the positive integer $N\geq 1$ either predefined or automatically determined according to different channel selection criteria.

• 2.

  A simple supervised time synchronization framework is proposed. It first picks the output of the best channel as a reference signal, then estimates the relative time delay of the other channels by a traditional time delay estimator, and finally synchronize the channels according to the estimation result, where the best channel is selected in a supervised manner.

• 3.

  A speech enhancement algorithm is implemented as an example. The algorithm applies the channel selection and time synchronization frameworks to deep beamforming. It is designed to demonstrate the overall effectiveness and flexibility of DAB. Its implementation is straightforward without a large modification of existing deep beamforming algorithms.

• 4.

  The effects of acoustic features on performance are studied. It is known that the performance of deep-learning-based speech enhancement relies heavily on acoustic features. In this paper, we further emphasize its importance on DAB by carrying out the first study on the effects of different handcrafted features on the performance. In this study, a variant of short term Fourier transform (STFT) and the common multi-resolution cochleagram (MRCG) features are used for comparison.

We have conducted an extensive experimental comparison between DAB and its deep-learning-based multichannel speech enhancement counterpart with linear microphone arrays, in scenarios where the speech sources and microphone arrays were placed randomly in typical physical spaces with random time delay, and the noise sources were either diffuse noise or point source noise. Experimental results with noise-independent training show that DAB outperforms its counterpart by a large margin. Its experimental conclusion is consistent across different hyperparameter settings and handcrafted features, though a good handcrafted feature do improve the overall performance.

The core idea of the proposed method has been released at [Zhang, 2018]. The main difference between the method here and [Zhang, 2018] is that we have added the time synchronization module, more channel selection algorithms and experiments on the point source noise environment here, which are incremental extensions that do not affect the fundamental claim of the novelty of the paper, compared to some related methods published after [Zhang, 2018].

This paper is organized as follows. Section 2 presents mathematical notations of this paper. Section 3 presents the signal model of ad-hoc microphone arrays. Section 4 presents the framework of the proposed DAB. Section 5 presents the channel selection module of DAB. Section 6 presents the application of DAB to speech enhancement. Section 7 evaluates the effectiveness of the proposed method. Finally, Section 8 concludes our findings.

We first introduce some notations here. Regular lower-case letters, e.g. $s$, $f$, and $\gamma$, indicate scalars. Bold lower-case letters, e.g. ${\mathbf{y}}$ and $\alpha$, indicate vectors. Bold capital letters, e.g. $\mathbf{P}$ and $\Phi$, indicate matrices. Letters in calligraphic fonts, e.g. $\mathcal{X}$, indicate sets. $\mathbf{0}$ ($\mathbf{1}$) is a vector with all entries being 1 (0). The operator ^T denotes the transpose. The operator ^H denotes the conjugate transpose of complex numbers.

Ad-hoc microphone arrays can significantly reduce the probability of the occurrence of far-field environments. We take the case described in Fig. 2 as an example. When a speaker and a microphone array are distributed randomly in a room, the distribution of the distance between the speaker and an ad-hoc microphone array has a smaller variance than that between the speaker and a conventional microphone array (Figs. 2a and 2b). For example, the conventional array has a probability of 24% to be placed over 10 meters away from the speech source, while the number regarding to the ad-hoc array is only 7%. Particularly, the distance between the best microphone in the ad-hoc array and the speech source is only 1.9 meters on average, and the probability of the distance that is larger than 5 meters is only 2% (Fig. 2c).

Here we build the signal model of an ad-hoc microphone array. All speech enhancement methods throughout the paper operate in the frequency domain on a frame-by-frame basis. Suppose that a physical space contains one target speaker, multiple noise sources, and an ad-hoc microphone array of $M$ microphones. The physical model for the signals arrived at the ad-hoc array is assumed to be

where $s(t,f)$ is the short-time Fourier transform (STFT) value of the target clean speech at time $t$ and frequency $f$, ${\mathbf{c}}(f)$ is the time-invariant acoustic transfer function from the speech source to the array which is an $M$-dimensional complex number:

$\mathbf{c}(f)s(t,f)$ and ${\mathbf{h}}(t,f)$ are the direct sound and early and late reverberation of the target signal, and ${\mathbf{n}}(t,f)$ is the additive noise:

which are the STFT values of the received signals by the $m$-th microphone at time $t$ and frequency $f$. Usually, we denote ${\mathbf{x}}(t,f)=\mathbf{c}(f)s(t,f)$.

After processed by the devices $\{D_{m}(\cdot)\}_{m=1}^{M}$ where the microphones are fixed, the signals that the DAB finally receives are:

with $\mathbf{z}(t,f)=[z_{1}(t,f),\ldots,z_{M}(t,f)]^{T}$. Real-world devices $\{D_{m}(\cdot)\}_{m=1}^{M}$ may cause many problems including the unsynchronization of time delay, clock rates, adaptive gain controllers, etc. Here we consider the time unsynchronization problem:

where $\tau_{m}$ is the time delay caused by the $m$th device.

A system overview of DAB is shown in Fig. 3. It contains three core components—a supervised channel selection framework, a supervised time synchronization framework, and a speech enhancement module.

The core idea of the channel selection framework is to filter the received signals ${\mathbf{z}}(t,f)$ by a channel-selection vector $\mathbf{p}=[p_{1},\ldots,p_{M}]^{T}$:

such that the channels that output low quality speech signals can be suppressed or even discarded, where $\mathbf{p}$ is the output mask of the channel-selection method described in the red box of Fig. 3, and $\circ$ denotes the element-wise product operator. Without loss of generality, we assume the selected channels are ${z}_{1}(t,f),\ldots,z_{N}(t,f)$.

The time synchronization module first selects the noisy signal from the best channel, assumed to be $z_{k}(t,f)$, as a reference signal by a supervised 1-best channel selection algorithm that will be described in Section 5.2. Then, it estimates the relative time delay of the noisy signals from the selected microphones over the reference signal by a time delay estimator:

where $h(z_{n}(t,f)|z_{k}(t,f))$ is the time delay estimator with $z_{k}(t,f)$ as the reference signal, and $\hat{\tau}_{n}$ is the estimated relative time delay of $z_{n}(t,f)$ over $z_{k}(t,f)$. Finally, it synchronizes the microphones according to the estimated time delay:

Note that $\hat{\tau}_{n}$ consists of the relative time delay caused by both the device and the signal transmission through air. Because developing a new accurate time delay estimator is not the focus of this paper, we simply use the classic generalized cross-correlation phase transform [Knapp & Carter, 1976, Carter, 1987] as the estimator, though many other time delay estimators can be adopted as well [Chen et al., 2006]. For example, the deep neural network based time delay estimators [Wang et al., 2018], which were original proposed for the estimation of the signal direction of arrival, can be adopted here too.

The speech enhancement module takes ${\mathbf{y}}(t,f)=[y_{1}(t,f),\ldots,y_{N}(t,f)]^{T}$ as its input. Many deep-learning-based speech enhancement methods can be used directly or with slight modification as the speech enhancement module. Here we take the MVDR based deep beamforming as an example. Because the deep beamforming is trained in a single-channel fashion as the other parts of DAB, it makes the overall DAB flexible in incorporating any number of microphones in the test stage without retraining or modifying the DAB model. This is an important requirement of real world applications that should also be considered in other DAB implementations. Note that, if $N=1$, then DAB outputs the noisy speech of the selected single channel directly without resorting to deep beamforming anymore.

In the following two sections, we will present the supervised channel selection framework and speech enhancement module respectively.

The channel-selection algorithm is applied to each channel independently. It contains two steps described in the following two subsections respectively.

After getting the synchronized signals ${\mathbf{y}}(t,f)=[y_{1}(t,f),\ldots,y_{N}(t,f)]^{T}$, we may use existing multichannel signal processing techniques directly or with slight modification for a specific application. Here we use a deep beamforming algorithm [Heymann et al., 2016, Wang & Wang, 2018] directly for speech enhancement as an example.

The deep beamforming algorithm finds a linear estimator ${\mathbf{w}}_{\rm{opt}}(f)$ to filter $\mathbf{y}(t,f)$ by the following equation:

where $\hat{x}_{\rm{ref.}}(t,f)$ is an estimate of the direct sound at the reference microphone of the array. For example, MVDR finds ${\mathbf{w}}_{\rm{opt}}$ by minimizing the average output power of the beamformer while maintaining the energy along the target direction:

where $\mbox{\boldmath{$\Phi$}}_{{\mathbf{n}}{\mathbf{n}}}(f)$ is an $M\times M$-dimensional cross-channel covariance matrix of the received noise signal ${\mathbf{n}}(f)$. (35) has a closed-form solution:

where the variables $\widehat{\mbox{\boldmath{$\Phi$}}}_{{\mathbf{n}}{\mathbf{n}}}(f)$ and $\hat{{\mathbf{c}}}(f)$ are the estimates of ${\mbox{\boldmath{$\Phi$}}}_{{\mathbf{n}}{\mathbf{n}}}(f)$ and ${{\mathbf{c}}}(f)$ respectively which are derived by the following equations according to [Zhang et al., 2017, Wang & Wang, 2018]:

where $\widehat{\mbox{\boldmath{$\Phi$}}}_{{\mathbf{x}}{\mathbf{x}}}(f)$ is an estimate of the covariance matrix of the direct sound ${\mathbf{x}}(t,f)$, ${\rm{principal}}(\cdot)$ is a function returning the first principal component of the input square matrix, and $\eta(t,f)$ and $\xi(t,f)$ are defined as the product of individual estimated T-F masks:

Note that, in our experiments, when we calculate $\eta(t,f)$ and $\xi(t,f)$, we take all channels of the ad-hoc array into consideration, which empirically results in slight performance improvement over the method that we take only the selected channels into the calculation.

In this section, we study the effectiveness of DAB in diffuse noise and point source noise environments under the situation where the output signals of the channels have random time delay caused by devices. Specifically, we first present the experimental settings in Section 7.1, then present the experimental results in the diffuse noise and point source noise environments in Section 7.2, and finally discuss the effects of hyperparameter settings on performance in Sections 7.3 and 7.4.

In this paper, we have proposed deep ad-hoc beamforming, which is to our knowledge the first deep learning method designed for ad-hoc microphone arrays.³³3This claim was made according to the fact that the core idea of the paper has been put on arXiv [Zhang, 2018] in January 2019. DAB has the following novel aspects. First, DAB employs an ad-hoc microphone array to pick up speech signals, which has a potential to enhance the speech signals with equally high quality in a range where the array covers. It may also significantly improve the SNR at the microphone receivers by physically placing some microphones close to the speech source in probability. Second, DAB employs a channel-selection algorithm to reweight the estimated speech signals with a sparsity constraint, which groups a handful microphones around the speech source into a local microphone array. We have developed several channel-selection algorithms as well. Third, we have developed a time synchronization framework based on time delay estimators and the supervised 1-best channel selection. At last, we emphasized the importance of acoustic features to DAB by carrying out the first study on how different acoustic features affect the performance.

Besides the above novelties and advantages, the proposed DAB is flexible in incorporating new development of DNN-based single channel speech processing techniques, since that its model is trained in the single-channel fashion. Its test process is also flexible in incorporating any number of microphones without retraining or revising the model, which meets the requirement of real-world applications. Moreover, although we applied DAB to speech enhancement as an example, we may apply it to other tasks as well by replacing the deep beamforming to other task-specific algorithms.

We have conducted extensive experiments in the scenario where the location of the speech source is far-field, random, and blind to the microphones. Experimental results in both the diffuse noise and point source noise environments demonstrate that DAB outperforms its MVDR based deep beamforming counterpart by a large margin given enough number of microphones. The conclusion is consistent across different acoustic features.

The research on DAB is only at the beginning. There are too many open problems. Here we list some urgent topics as follows. (i) How to synchronize microphones when the clock rates and power amplifiers at different devices are different. (ii) How to design new spatial acoustic features for ad-hoc microphone arrays beyond interaural time difference or interaural level difference. (iii) How to design a model that can be trained with multichannel data collected from ad-hoc microphone arrays, and generalize well to fundamentally different ad-hoc microphone arrays in the test stage. (iv) How to handle a large number of microphones (e.g. over 100 microphones) for a large room that contains many complicated acoustic environments.

The author would like to thank Prof. DeLiang Wang for helpful discussions.

This work was supported in part by the National Key Research and Development Program of China under Grant No. 2018AAA0102200, in part by National Science Foundation of China under Grant No. 61831019, 61671381, in part by the Project of the Science, Technology, and Innovation Commission of Shenzhen Municipality under grant No. JCYJ20170815161820095, and in part by the Open Research Project of the State Key Laboratory of Media Convergence and Communication, Communication University of China, China under Grant No. SKLMCC2020KF009.