Research map / Vision-language and multimodal / Cross-modal learning
RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RTFS-Net, a time-frequency domain audio-visual speech separation network that models time and frequency axes with recurrent units in a compressed subspace, achieving SOTA-level separation with under 1M parameters and lower MACs than prior time-domain methods.
Problem
Audio-visual speech separation has been dominated by heavy time-domain models; prior time-frequency (TF) domain AVSS methods underperform, and existing ones are computationally expensive or neglect complex-valued STFT structure and multi-scale visual fusion.
Approach
Process complex STFT bins with dual-path (frequency then time) SRU recurrence in a compressed subspace, fuse visual cues with a cross-dimensional attention fusion block, reconstruct with attention-based TF-AR units, and extract the target via a complex-multiplication mask (S^3) instead of a real-valued mask.
Key results
- RTFS-Net-12 reaches 14.9 dB SI-SNRi on LRS2-2Mix (vs. CTCNet 14.3) with 0.7M parameters and 56.4G MACs versus CTCNet's 7.0M and 167.2G.
- RTFS-Net-12 achieves 17.5 dB SI-SNRi on LRS3-2Mix and 12.4 dB on VoxCeleb2-2Mix, exceeding CTCNet's 17.4 and 11.9.
- RTFS-Net-20 reaches 15.4 dB SI-SNRi on LRS2-2Mix, outperforming CTCNet by 1.1 dB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase RTFS Blocks from 4 to 12 | RTFS-Net · LRS2-2Mix | SI-SNRi: 14.1 -> 14.9 | Helped |
| Use CAF instead of adapted CTCNet fusion (ablation) | reduced RTFS-Net-4 · LRS2-2Mix | SI-SNRi: 11.3 -> 11.7 | Helped |
| Add TF-AR units instead of interpolation and addition (ablation) | RTFS-Net-4 · LRS2-2Mix | SI-SNRi: 13.0 -> 14.1 | Helped |
| Use complex-valued S³ extraction instead of regression (ablation) | reduced RTFS-Net · LRS2-2Mix | SI-SNRi: 10.0 -> 11.3 | Helped |
| Use complex-valued S³ extraction instead of mask extraction (ablation) | reduced RTFS-Net · LRS2-2Mix | SI-SNRi: 10.8 -> 11.3 | Helped |
| Use complex-valued S³ extraction instead of mask plus depth-wise gate (ablation) | reduced RTFS-Net · LRS2-2Mix | SI-SNRi: 10.8 -> 11.3 | Helped |
| Use complex-valued S³ extraction instead of mask plus gate (ablation) | reduced RTFS-Net · LRS2-2Mix | SI-SNRi: 11.1 -> 11.3 | Helped |
| Share RTFS Block parameters (ablation) | RTFS-Net-4 · LRS2-2Mix | SI-SNRi: 13.6 -> 13.7 | Helped |
| Increase compression scale q from 2 to 3 (ablation) | RTFS-Net-4 · LRS2-2Mix | SI-SNRi: 14.1 -> 12.8 | Hurt |
| Replace SRUs with MHSA (ablation) | RTFS-Net-4 · LRS2-2Mix | SI-SNRi: 14.1 -> 13.1 | Hurt |
| Increase RTFS Blocks from 12 to 20 (ablation) | RTFS-Net · LRS2-2Mix | SI-SNRi: 14.9 -> 15.4 | Helped |
| Use SRUs instead of LSTMs (ablation) | RTFS-Net-4 · LRS2-2Mix | SI-SNRi: 14.0 -> 14.1 | No clear effect |
Surprising findings
- RTFS-Net-4 with 0.7M params is nearly on par with CTCNet on LRS2-2Mix (14.1 vs 14.3 dB SI-SNRi), but on VoxCeleb2-2Mix the gap is larger for smaller configs.
- Transformer-based (MHSA) replacements for the SRU blocks lost about 1 dB SI-SNRi despite fewer parameters.
Limitations
- Speaker-separation performance still trails CTCNet on some configs (e.g., RTFS-Net-4 and -6 on VoxCeleb2-2Mix), and the paper says it does not focus on SOTA.
- Inference time of the proposed model is not always lower than baselines (e.g., AVLIT is faster at 53.4 ms), and the SRU unfolding increases input dimensions significantly.
- Evaluation of downstream speech recognition relies on the Google Speech-to-Text API, which is a non-reproducible commercial service.
Experimental setup
Datasets: LRS2-2Mix LRS3-2Mix VoxCeleb2-2Mix
Benchmarks: LRS2-2Mix LRS3-2Mix VoxCeleb2-2Mix LRS2-3Mix
Metrics: SI-SNRi SDRi PESQ Params MACs inference time WER
Baselines: CTCNet AV-ConvTasNet VisualVoice AVLIT CaffNet-C Thanh-Dat
Builds on: CTCNet TDANet TF-GridNet Simple Recurrent Unit DPTNet
Closest papers by meaning
- TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation (ICLR 2025 · similarity 0.87)
- Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation (NeurIPS 2024 · similarity 0.87)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.85)
- video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models (ICML 2024 · similarity 0.85)
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.84)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.