One-sentence Summary

Researchers from the University of Science and Technology of China and Georgia Institute of Technology propose a deep neural network (DNN)-based regression framework for speech enhancement that learns a mapping from noisy to clean speech, leveraging a large training set, global variance equalization, dropout, and noise-aware training to improve generalization, and show that it significantly outperforms conventional minimum mean square error (MMSE) methods by effectively suppressing non-stationary noise without musical artifacts in real-world scenarios.

Key Contributions

• A supervised speech enhancement framework uses a deep neural network as a nonlinear regression function to map noisy speech features to clean ones, trained on a large multi-condition dataset with over 100 noise types and extended acoustic context to improve generalization.
• Three techniques are introduced to further improve the DNN-based system: global variance equalization to reduce over-smoothing, dropout regularization to prevent overfitting, and noise-aware training to enhance robustness to unseen noise conditions.
• Experimental results demonstrate that the proposed approach achieves significant objective and subjective improvements over conventional MMSE-based methods, effectively suppresses non-stationary noise, and avoids musical artifacts on real-world recordings, even for unseen languages and recording environments.

Introduction

The authors address single-channel speech enhancement, a critical task for mobile communication, hearing aids, and robust speech recognition, where prior methods often struggle with non-stationary noise and produce artifacts like musical noise or over-smoothing. Traditional MMSE-based estimators and shallow neural networks suffer from limited noise tracking and poor generalization to unseen acoustic conditions, while binary mask approaches improve intelligibility but sacrifice speech quality. The authors extend a regression DNN framework by training on a large, heterogeneous set of over 100 noise types and leveraging full acoustic context (broad frequency band and context frames) to reduce discontinuity. They further introduce three strategies: global variance equalization to counteract over-smoothing, dropout to improve generalization, and noise-aware training, yielding a robust enhancer that maintains quality across diverse unseen noise environments.

Method

The authors propose a DNN-based speech enhancement framework that operates in two main stages: training and enhancement. As shown in the figure below:

In the training stage, a DNN-based regression model is trained using log-power spectral features extracted from pairs of noisy and clean speech data. Short-time Fourier analysis is applied to compute the discrete Fourier transform (DFT) of overlapping windowed frames, followed by log-power spectra calculation. In the enhancement stage, the well-trained DNN processes noisy speech features to predict clean speech features. Once the estimated log-power spectral features $\hat{X}^l(d)$X^l(d) are obtained, the reconstructed spectrum $\hat{X}^f(d)$X^f(d) is computed as:

$\hat{X}^f(d) = \exp\{\hat{X}^l(d) / 2\} \exp\{j \angle Y^f(d)\}$X^f(d)=exp{X^l(d)/2}exp{j∠Yf(d)}

where $\angle Y^f(d)$∠Yf(d) is the phase of the noisy speech. The phase is extracted directly from the noisy signal, as estimating it is difficult at low SNRs. Finally, an inverse DFT (IDFT) and overlap-add method synthesize the waveform.

The architecture is a feed-forward neural network mapping noisy features to clean features. The training procedure consists of unsupervised pre-training and supervised fine-tuning. The training process is illustrated in the figure below:

To avoid local minima, the authors first pre-train a deep generative model by stacking multiple restricted Boltzmann machines (RBMs). Since input features are real-valued, the first RBM is a Gaussian-Bernoulli RBM, followed by Bernoulli-Bernoulli RBMs trained layer-by-layer using contrastive divergence. Subsequently, back-propagation with a mean squared error (MMSE) objective function fine-tunes all parameters. The error function is:

$Er = \frac{1}{N} \sum_{n=1}^{N} \| \hat{\mathbf{X}}_n(\mathbf{Y}_{n-\tau}^{n+\tau}, \mathbf{W}, \mathbf{b}) - \mathbf{X}_n \|_2^2$Er=N1​∑n=1N​∥X^n​(Yn−τn+τ​,W,b)−Xn​∥22​

where $\hat{\mathbf{X}}_n$X^n​ and $\mathbf{X}_n$Xn​ are estimated and reference features, $N$N is mini-batch size, and $\mathbf{Y}_{n-\tau}^{n+\tau}$Yn−τn+τ​ is the noisy feature vector with context window $2\tau+1$2τ+1. Parameters are updated iteratively using stochastic gradient descent with momentum and weight decay.

A common issue in DNN-based enhancement is over-smoothing, which causes a muffling effect. The global variance of estimated features is typically smaller than that of reference features. This discrepancy is visualized in the figure below:

This over-smoothing suppresses formant peaks, particularly in high frequencies, leading to muffled speech. The resulting over-smoothing effect is evident in the figure below:

To address this, the authors propose global variance equalization. A dimension-dependent equalization factor $\alpha(d)$α(d) is defined as:

$\alpha(d) = \sqrt{\frac{GV_{ref}(d)}{GV_{est}(d)}}$α(d)=GVest​(d)GVref​(d)​​

where $GV_{ref}(d)$GVref​(d) and $GV_{est}(d)$GVest​(d) are the global variances of reference and estimated features. A dimension-independent factor $\beta$β can also be used. The DNN output $\hat{X}(d)$X^(d) is transformed back and scaled by the equalization factor $\eta$η (either $\alpha$α or $\beta$β) to sharpen formant peaks and suppress residual noise:

$\hat{X}''(d) = \hat{X}(d) * \eta * v(d) + m(d)$X^′′(d)=X^(d)∗η∗v(d)+m(d)

where $m(d)$m(d) and $v(d)$v(d) are the mean and variance of input noisy features.

To improve generalization to unseen noise types, the authors employ dropout training. During training, a percentage $\rho$ρ of neurons in the input and hidden layers are randomly omitted. This prevents complex co-adaptations and over-fitting. At the enhancement stage, weights are discounted by $(1-\rho)$(1−ρ).

Furthermore, to enable noise awareness, the DNN input vector is augmented with an estimate of the noise $\hat{\mathbf{Z}}_n$Z^n​, calculated by averaging the first $T$T frames of the utterance:

$\mathbf{V}_n = [\mathbf{Y}_{n-\tau}, \dots, \mathbf{Y}_n, \dots, \mathbf{Y}_{n+\tau}, \hat{\mathbf{Z}}_n]$Vn​=[Yn−τ​,…,Yn​,…,Yn+τ​,Z^n​]

$\hat{\mathbf{Z}}_n = \frac{1}{T} \sum_{t=1}^{T} \mathbf{Y}_t$Z^n​=T1​∑t=1T​Yt​

This allows the DNN to utilize online noise information for better prediction. Dropout is also applied to the estimated noise spliced into the input layer to compensate for noise spectrum variability.

Experiment

The evaluation setup used a multi-condition training set of 2500 hours with 104 environmental noise types and TIMIT clean speech, testing on 15 unseen noise types and real-world recordings. Experiments showed that deeper networks, longer acoustic context, and more diverse training data improved generalization, while the proposed dropout, global variance equalization, and noise adaptive training strategies were complementary and significantly outperformed the LogMMSE baseline, especially under non-stationary noise. Subjective listening tests confirmed a strong preference for DNN-enhanced speech, and the approach effectively suppressed burst and non-stationary noise even in cross-language real-world scenarios.

Training with 104 noise types instead of just 4 consistently improves both perceptual quality (PESQ) and reduces log-spectral distortion (LSD) across all tested signal-to-noise ratios. The LSD benefit is especially pronounced in very noisy conditions. At every SNR, the model exposed to 104 noise types yields higher PESQ scores than the model trained with only 4 noise types. The LSD reduction from using more noise types grows substantially as SNR decreases, with the largest improvement occurring at the lowest SNR tested.

Deep neural network speech enhancement yields consistently higher PESQ scores than both noisy baselines and shallow SNN models across all tested SNR levels and unseen noise types. The best DNN variants achieve average PESQ values near 2.60, substantially above the noisy average of 2.09, and the quality gain remains prominent even at very low SNRs. All DNN-based methods outperform the two SNN models and the unprocessed noisy speech at every SNR condition. The top-performing DNN configurations (DNN_3 and DNN_4) reach average PESQ scores of 2.59–2.60, compared to 2.09 for noisy speech and 2.48 for the best SNN. From 20 dB down to -5 dB SNR, the leading DNN maintains a clear advantage, with absolute PESQ improvements over noisy speech ranging from 0.38 to 0.49 points.

The proposed normalized clean log-power spectra training target achieves consistently higher PESQ scores than the IRM and FFT-mask targets across all tested unseen noise environments and SNR levels. The IRM and FFT-mask targets yield similar performance, both trailing the proposed target, with the advantage of the proposed target most evident at higher SNRs. PESQ declines monotonically with decreasing SNR for all training targets. The proposed target obtains the highest PESQ in every condition; for example, at SNR20 in the Destroyer engine environment it scores 3.51, compared to 3.43 for IRM and 3.41 for FFT-mask. The PESQ gap between the proposed target and the other two targets is larger at high SNRs (e.g., SNR20 and SNR15) and narrows at very low SNRs such as SNR-5.

RBM pre-training and random initialization yield nearly identical speech enhancement results across all tested SNRs and unseen noise environments. For PESQ, LSD, and SSNR, the differences are negligible, indicating that the pre-training strategy offers no meaningful advantage over random weight initialization when using 100 hours of training data. PESQ scores differ by at most 0.02 between RBM pre-training and random initialization at every SNR level. LSD and SSNR values remain virtually unchanged, with no consistent improvement from pre-training.

Across three unseen noise environments, NAT techniques consistently improve PESQ over the DNN baseline, with the β variant giving the highest scores at every SNR. The Destroyer engine condition yields the best speech quality, while the HF channel is the most challenging. Gains are robust from -5 dB to 20 dB SNR. The β NAT variant outperforms both the α variant and the vanilla DNN baseline in all tested noise conditions and SNR levels. Speech quality is highest in the Destroyer engine noise and lowest in the HF channel noise, with the DNN baseline achieving higher PESQ in engine noise than in the HF channel at every SNR.

The experiments evaluate speech enhancement across multiple conditions, including training with 4 versus 104 noise types, DNN versus shallow SNN architectures, alternative training targets, pre-training strategies, and noise adaptation techniques, using PESQ, LSD, and SSNR metrics over SNR levels from -5 dB to 20 dB in unseen noise environments. Training with 104 noise types, deeper DNN models, and a normalized log-power spectra target consistently improve perceptual quality, with the β noise adaptation variant further boosting performance. RBM pre-training provides no advantage over random initialization when using 100 hours of training data. The best configurations achieve substantial gains over noisy baselines, with the largest improvements at very low SNRs.