One-sentence Summary

Researchers from Qinghai University, Tsinghua University, and the Chinese Institute for Brain Research release RealSSA, a realistic audio-visual target speaker extraction benchmark with near-field and far-field interference, and propose Falcon, a causal method that combines multi-scale time-frequency separation with an encoder-space speaker contrastive loss to achieve state-of-the-art extraction quality while reducing parameters by $91.9\%$91.9%, computation by $20.2\times$20.2×, and GPU inference latency by $11.1\times$11.1× compared with causal AV-TFGridNet.

Key Contributions

• RealSSA is introduced as a realistic audio-visual target speaker extraction benchmark with RealSSA-Sim and RealSSA-Real. RealSSA-Sim provides controllable simulated scenes with 1 to 5 dynamic near-field interfering speakers, far-field scenes with 10 to 15 far-field interferers, reverberation, and sudden noise, while RealSSA-Real provides a held-out real-recorded evaluation set.
• Falcon is a causal extraction method whose separator performs multi-scale time-frequency separation in a single encoder-decoder pass. Falcon is trained with an encoder-space speaker contrastive loss that suppresses near-field leakage at zero inference cost and transfers across backbones.
• Falcon achieves state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix, and competitive perceptual quality, WER, and target-speaker similarity on RealSSA-Real. Compared with causal AV-TFGridNet, it reduces parameters by 91.9%, computation by 20.2×, and GPU inference latency by 11.1×.

Introduction

Audio-visual target speaker extraction (AVTSE) uses a speaker’s lip stream as an enrollment cue to isolate that speaker from a multi-speaker mixture, and latency-sensitive applications such as video conferencing and hearing assistive devices require causal processing. Prior causal AVTSE work is limited in two ways: benchmarks fix speaker counts and acoustic conditions, with no real-recorded evaluation, while causal architectures are either too compute-heavy for edge deployment or reuse small modules iteratively in ways that erode latency gains. The authors introduce RealSSA, a benchmark with simulated scenes spanning 1 to 5 dynamic near-field interferers, optional 10 to 15 far-field interferers, joint reverberation, ambient noise, and sudden noise, plus a held-out real-recorded evaluation set. They also propose Falcon, a single-pass causal model whose decoder applies frequency-axis, time-axis, and joint time-frequency modeling at different levels, along with an encoder-space speaker contrastive loss that reduces near-field leakage at no inference cost. Falcon matches or exceeds strong causal baselines while reducing parameters by 91.9%, computation by 20.2×, and GPU inference latency by 11.1× compared with causal AV-TFGridNet.

Dataset

The authors use RealSSA as the main dataset, with two complementary subsets:

• RealSSA-Sim: a simulated scene suite used for training, validation, and testing.

  • Contains four acoustic scenes: small meeting rooms, medium indoor spaces, large public spaces, and outdoor environments.
  • Near-field interfering speakers vary from 1 to 5 per sample; scenes with far-field interference also include 10 to 15 far-field interfering speakers. A speaker is near-field when located within 3 m of the microphone, otherwise far-field.
  • Target audio-visual data are sampled from LRS3; interfering speakers come from LibriSpeech; continuous background noise comes from WHAM!; sudden noise events come from Hive.
  • Room simulation uses gpuRIR for room impulse responses, with scene type, room geometry, and RT60 sampled per sample.
  • Mixing normalizes target speech to -23 LUFS, adds interferers and noise at scene-specific SIR ranges, peak-limits the final mixture to -1 dBFS, and outputs single-channel 16 kHz audio.
  • Size: 100,826 training, 9,133 validation, and 9,169 test samples, totaling about 153 hours. The four scene types are uniformly sampled, with an average of 2.5 near-field interfering speakers per sample.

• RealSSA-Real: a held-out real-world evaluation set.

  • Recorded with a single mobile device in everyday indoor, public, and outdoor environments, with natural background noise, room reverberation, and uncontrolled interfering speech.
  • Used exclusively for evaluation and never used for training or validation.
  • Post-processed only to match the AVTSE input format: audio resampled to 16 kHz, video converted to 25 fps, and 96x96 grayscale mouth regions extracted.
  • Evaluated with DNSMOS and NISQA for perceptual quality, WER for speech recognizability, and speaker similarity for target-speaker consistency.

For the paper's experiments, all datasets use 16 kHz audio paired with 96x96 grayscale mouth crops at 25 fps. Public benchmarks LRS2, LRS3, and VoxCeleb2 follow their official partitions. RealSSA-Sim is used for training, validation, and testing, while RealSSA-Real is used only for evaluation.

Method

The authors propose Falcon, a model designed to extract target speech from a mixture waveform given a mouth video of the target speaker. The mixture is modeled as the sum of target speech, near-field interfering speech, far-field interfering speech, and additive environmental noise.

As shown in the figure below:

The visual backbone, frozen during training, extracts mouth features from the video and maps them through a visual bottleneck. The audio encoder applies STFT to the mixture, concatenates the real, imaginary, and magnitude spectra, and processes them through a 2D convolutional layer. Before fusion, the audio feature is processed by an AudioBlock, and the visual feature is processed by a VideoBlock that aggregates temporal context and upsamples to the audio frame rate. The visual feature is expanded along the frequency axis and added to the audio feature to produce a fused feature. This fused feature is then processed by a Separator to obtain the target-speaker feature. A mask generator predicts a time-frequency mask from this feature. The channel dimensions of the mask and the initial audio encoding are split into real and imaginary sub-bands, and the masked embedding is obtained via complex multiplication:

Finally, an audio decoder maps the masked embedding back to spectral components and reconstructs the target speech via iSTFT.

The Separator plays a central role in extracting the target speaker features. It is designed as an asymmetric U-Net where time-frequency modeling is concentrated at the encoder bottleneck to form a coarse scene context, while level-specific blocks at every decoder resolution recover fine spectral structure.

As shown in the figure below:

The input feature first passes through a depthwise convolution to form a residual branch, which serves as a global skip connection. A pointwise convolution projects this to the hidden channel dimension, followed by a streaming-causal layer normalization. The encoder consists of four levels of depthwise convolutions that progressively halve both time and frequency axes. Time-frequency modeling is applied only at the bottleneck using a Time-Frequency Refinement Block (TFRB).

As shown in the figure below:

Each TFRB cascades a bidirectional LSTM along the frequency axis, a unidirectional LSTM along the time axis, and a 2D self-attention mechanism operating on the flattened time-frequency plane with a block-causal mask.

To inject global context into the decoder, a Global Context Injection (GCI) module is used. It upsamples the bottleneck feature and uses a gating mechanism to emphasize or suppress frequency bands of the local skip connections. The decoder restores resolution coarse-to-fine, applying TFRBs at different resolutions to span four time-frequency scales in a single forward pass. The final decoder output is projected and added to the residual branch to yield the target speaker feature.

To explicitly push the estimated speech away from near-field interferers, the authors introduce a contrastive loss that reuses the audio encoder. The query is the estimated target in encoder space, while the positive and negative references are the clean target and the near-field interferers, respectively, passed through the audio encoder with stop-gradient. Far-field interferers are excluded from the negative set. A lightweight speaker pooling function maps each encoder-space feature to a unit-norm speaker embedding. The contrastive loss combines a positive attraction term with a softmax-weighted, margin-based repulsion over negatives:

where $m=0.5$m=0.5 is the angular margin and $\tau$τ is the inverse temperature, which is linearly annealed from 0 to 1 during training. The final training objective combines this contrastive loss with a standard reconstruction loss based on negative SI-SNR.

Experiment

The evaluation uses three public AVTSE benchmarks plus the simulated and real RealSSA sets, measuring separation quality, perceptual quality, intelligibility, speaker similarity, and computational efficiency. Falcon achieves the best SI-SNRi, SDRi, and perceptual quality while also attaining the lowest WER and highest target-speaker similarity on key extraction tasks, with large reductions in parameters, MACs, and inference latency compared with strong baselines. Generalization experiments show that training on RealSSA-Sim improves real-world perceptual quality, and ablations confirm the benefits of multi-stage downsampling, encoder-space speaker contrastive loss, and temperature annealing.

Existing AVTSE benchmarks are limited to fixed near-field speaker pairs and offer little control over acoustic conditions; only a subset adds environmental noise, and none include far-field sources, controlled reverberation, sudden noise, or real recorded evaluation. RealSSA relaxes these constraints by supporting variable speaker counts and adding far-field sources, controlled reverberation, environmental noise, sudden noise, and real recorded evaluation. Prior benchmarks such as LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix fix near-field speaker counts and do not provide controlled reverberation or sudden noise. RealSSA is the only benchmark in this comparison that includes far-field speakers, variable speaker counts, controlled reverberation, sudden noise, and real recorded evaluation.

The simulated suite defines four acoustic scene types that span small indoor rooms, medium and large indoor spaces, and outdoor environments. Indoor scenes progress from short reverberation and only near-field interferers to longer reverberation with many far-field interferers, while outdoor scenes pair minimal reverberation with stronger background noise. Sudden noise events and source distances also vary across the scene types. Reverberation time increases across small, medium, and large indoor scenes, while outdoor scenes remain almost anechoic. Small meeting rooms include only near-field interfering speakers, whereas the other three scenes add far-field interfering speakers at greater distances. The environmental noise SIR range shifts downward from indoor scenes to outdoor scenes, indicating progressively stronger background noise. Sudden noise events occur least often in small meeting rooms and most often in large public scenes.

Falcon achieves the highest SI-SNRi and SDRi on all four simulated benchmarks, with only a slim RealSSA-Sim edge over AV-TF-GridNet. On real recordings, Falcon reaches the best DNSMOS while tying SwiftNet-12 and outperforming AV-TF-GridNet and Dolphin. Among the displayed baseline rows, AV-TF-GridNet is competitive on RealSSA-Sim but weaker on VoxCeleb2-2Mix, where RTFSNet-12 leads. Falcon leads all simulated benchmarks in SI-SNRi and SDRi, with its RealSSA-Sim result slightly ahead of AV-TF-GridNet. On RealSSA-Real, Falcon ties SwiftNet-12 for the highest DNSMOS and clearly exceeds AV-TF-GridNet and Dolphin.

Falcon achieved the lowest word error rate on the simulated set among extraction systems and the highest target-speaker similarity on both evaluation sets. On real recordings, Falcon's word error rate ranked second, trailing the best baseline by a small margin. The results indicate a favorable balance between transcription accuracy and target-speaker consistency relative to prior systems. Falcon achieved the best simulated-set WER among extraction systems, outperforming the strongest baseline on that set. Falcon produced the highest target-speaker similarity on both simulated and real recordings, surpassing the next-best baseline on each. On real recordings, Falcon's WER placed second, behind AV-Mossformer2.

Falcon achieves the lowest computational and memory costs among the compared causal models, matching Swift-Net-12 for parameter count while requiring substantially fewer MACs and lower GPU latency. It also reduces CPU latency, RAM, and GPU inference memory relative to all listed baselines. Falcon matches Swift-Net-12 in parameter count but needs about a quarter of its MACs and under a third of its GPU latency. Compared with the strongest baseline, Falcon attains higher separation quality while using substantially fewer parameters, MACs, and GPU latency. Falcon reports the lowest CPU latency, GPU latency, RAM, and GPU inference memory among all compared methods.

The experiments evaluate a new RealSSA benchmark with variable far-field speakers, controlled reverberation, environmental and sudden noise, and real recordings. Falcon is compared against baselines across simulated indoor and outdoor scenes and real data, validating separation quality, transcription accuracy, speaker similarity, and efficiency. It consistently leads simulated separation metrics and real DNSMOS, achieves the best simulated word error rate and highest target-speaker similarity, and shows the lowest computational and memory cost among causal models.