Research map / Architectures and efficiency / Attention and transformers
Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SepReformer, an asymmetric encoder-decoder speech separator that splits features early into speaker streams processed by a weight-shared decoder, plus global/local Transformer blocks that replace dual-path chunking.
Problem
Speech separation in the time domain suffers from a late-split bottleneck where one feature sequence must encode all speakers, and dual-path models for long sequences are computationally expensive.
Approach
Expand the feature sequence by the number of speakers early (early split) and process the separated streams with a weight-shared decoder plus cross-speaker attention; replace dual-path chunking with global (EGA) and local (CLA) Transformer blocks operating on full sequences, with a multi-stage SI-SNR loss.
Key results
- SepReformer-B reaches 23.8 dB SI-SNRi on WSJ0-2Mix with 14.2M parameters and 39.8 G/s MACs, outperforming TF-GridNet's 23.5 dB at 460.8 G/s.
- SepReformer-L with dynamic mixing reaches 25.1 dB SI-SNRi on WSJ0-2Mix, 18.4 dB on WHAM! and 17.2 dB on WHAMR!, reported as state-of-the-art in the paper.
- SepReformer-T (3.5M params, 10.4 G/s) reaches 22.4 dB SI-SNRi on WSJ0-2Mix, exceeding Sepformer's 20.4 dB at about 10x fewer MACs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace late split and original decoder with early split and a shared decoder (ablation) | SepRe method · WSJ0-2Mix | SI-SNRi: 19.0 -> 21.3 | Helped |
| add cross-speaker processing to early split with a shared decoder (ablation) | SepRe method · WSJ0-2Mix | SI-SNRi: 21.3 -> 22.4 | Helped |
| add multi-loss to early split, shared decoder, and CS (ablation) | SepRe method · WSJ0-2Mix | SI-SNRi: 22.6 -> 23.8 | Helped |
| add multi-loss to early split with a shared decoder, without CS (ablation) | SepRe method · WSJ0-2Mix | SI-SNRi: 22.4 -> 23.1 | Helped |
| apply ESSD, CS, and multi-loss (ablation) | Conv-TasNet · WSJ0-2Mix | SI-SNRi: 15.3 -> 19.5 | Helped |
| apply ESSD, CS, and multi-loss (ablation) | Sepformer · WSJ0-2Mix | SI-SNRi: 20.4 -> 22.7 | Helped |
| use encoder-decoder block counts (2,3) rather than (3,2) or (4,1) (ablation) | SepReformer-B · WSJ0-2Mix | SI-SNRi: 22.8 -> 23.8; 21.7 -> 23.8 | Helped |
| add the linear-layer and sigmoid gate to EGA (ablation) | SepReformer-B · WSJ0-2Mix | SI-SNRi: 23.2 -> 23.8 | Helped |
| replace standard FFN with Dconv- and GLU-based GCFN (ablation) | SepReformer-B · WSJ0-2Mix | SI-SNRi: 23.0 -> 23.8 | Helped |
| replace standard FFN with FFN plus Dconv (ablation) | SepReformer-B · WSJ0-2Mix | SI-SNRi: 23.0 -> 23.4 | Helped |
| replace standard FFN with FFN using GLU (ablation) | SepReformer-B · WSJ0-2Mix | SI-SNRi: 23.0 -> 23.3 | Helped |
| use the global-local Transformer with U-Net multi-scale processing instead of stacked global-local blocks (ablation) | Long-sequence model · WSJ0-2Mix | MACs (G/s): 43.1 -> 18.3 | Helped |
Surprising findings
- Sharing decoder weights after the early split improved over separate per-speaker decoders (21.3 vs 19.8 dB for tiny models), even without speaker information.
- For the large model, a weight-shared speaker split layer performed comparably to multiple split layers on WSJ0-2Mix, but multiple layers were better on WHAM! and WHAMR!.
Limitations
- Evaluated only on two-speaker mixtures; multi-speaker scenarios are not validated.
- Assumes the number of speakers is known in advance.
- Underlying mechanism of the SepRe method is not fully explained.
Experimental setup
Datasets: WSJ0-2Mix WHAM! WHAMR! Libri2Mix LibriSpeech LibriCSS
Benchmarks: WSJ0-2Mix WHAM! WHAMR! Libri2Mix LibriCSS
Metrics: SI-SNRi SDRi MACs parameters PESQ eSTOI WER
Baselines: Conv-TasNet DPRNN DPTNet Sepformer SuDoRM-RF TDANet MossFormer TF-GridNet QDPN WaveSplit ISCIT SFSRNet
Builds on: TasNet Conv-TasNet DPRNN Sepformer QDPN
Closest papers by meaning
- UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures (NeurIPS 2023 · similarity 0.89)
- Filter-Recovery Network for Multi-Speaker Audio-Visual Speech Separation (ICLR 2023 · similarity 0.89)
- Voice Separation with an Unknown Number of Multiple Speakers (ICML 2020 · similarity 0.88)
- RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation (ICLR 2024 · similarity 0.87)
- TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation (ICLR 2025 · similarity 0.86)
- Zipformer: A faster and better encoder for automatic speech recognition (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.