Research map / Large language models / Language, safety and interpretability
Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sortformer, an encoder-based speaker diarizer trained with a sort-based loss (alone or hybrid with PIL) that yields arrival-time-ordered speaker labels, and uses it to supervise speaker tags in a multi-speaker Canary ASR model trained with standard cross-entropy.
Problem
Permutation ambiguity in speaker labels makes PIL-based end-to-end diarization hard to integrate into ASR/LLM training, which needs standard token-level cross-entropy and speaker-annotated multi-speaker data that is scarce.
Approach
Sort Loss supervises speaker presence outputs in arrival-time order so speaker tokens in transcripts are also arrival-time sorted; the diarizer's sorted outputs are injected into ASR encoder states via sinusoidal speaker kernels, letting multi-speaker ASR train with plain cross-entropy.
Key results
- Sortformer trained with hybrid loss (alpha=0.5) reaches 5.87% DER on CALLHOME-part2 2-speaker, beating the PIL-only Sortformer (6.94%) and comparable to WavLM+EEND-VC-style baselines.
- On LibriSpeechMix, Sortformer-MS-Canary (293M) gets 4.61% WER on 2-mix and 9.05% on 3-mix versus 6.55% and 12.14% for MS-Canary without speaker supervision.
- On real multi-speaker recordings, the adapter-based Canary-1B system with word-level sorted speaker tokens reduces cpWER on AMI test to 26.71% from a 32.94% baseline-style no-supervision system.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine Sort Loss with PIL instead of using PIL alone (ablation) | Sortformer · DH3 and CALLHOME-part2 | DER: 18.33 -> 16.28; 7.28 -> 6.49; 11.57 -> 10.01; 18.80 -> 14.14 | Helped |
| Replace PIL-only training with Sort Loss-only training (ablation) | Sortformer · CALLHOME-part2 and CH109 | DER: 7.28 -> 7.42; 11.57 -> 12.68; 18.80 -> 19.42; 5.66 -> 9.08 | Mixed |
| Apply timestamp postprocessing (ablation) | Sortformer-Hybrid-Loss · DH3, CALLHOME-part2, CH109 | DER: 16.28 -> 14.76; 6.49 -> 5.87; 10.01 -> 8.46; 14.14 -> 12.59; 6.27 -> 6.86 | Mixed |
| Use frozen Sortformer speaker supervision rather than no speaker supervision (ablation) | Sortformer-MS-Canary System 2 · AMI-test | cpWER: 32.94% -> 28.17% | Helped |
| Fine-tune Sortformer rather than keeping it frozen (ablation) | Sortformer-MS-Canary · AMI-test | cpWER: 28.17% -> 32.74% | Hurt |
| Use ground-truth speaker labels for training supervision instead of Sortformer supervision (ablation) | Sortformer-MS-Canary System 4 · AMI-test | cpWER: 28.17% -> 26.83% | Helped |
| Use word-level rather than segment-level speaker-token objectives (ablation) | Sortformer-MS-Canary with adapter 256 · AMI-test and CH109 | WER: 18.58% -> 18.04%; 17.74% -> 16.46% | Helped |
| Add Sortformer speaker supervision | MS-Canary · LibriSpeechMix 2mix and 3mix | WER: 6.55 -> 4.61; 12.14 -> 9.05 | Helped |
| Add Sortformer speaker supervision | MS-Canary · LibriSpeechMix 1mix | WER: 2.19 -> 2.26 | Hurt |
Surprising findings
- Sort Loss alone achieves diarization performance comparable to PIL-trained model, and sort-only training yields only minimal training overhead.
- Segment-level speaker objective with the adapter approach degraded results compared with word-level objectives across settings.
Limitations
- Arrival-time estimation is error-prone, especially as the number of speakers grows, so pure Sort Loss needs PIL support.
- Limited to up to four speakers; streaming and target-speaker ASR are left to future work.
- Word timestamps are approximated syllable-wise when missing, which may be imprecise.
Experimental setup
Datasets: Fisher English AMI DIHARD3 VoxConverse ICSI AISHELL-4 NIST SRE 2000 CALLHOME LibriSpeech NIST SRE04-10 DipCo LibriSpeechMix CH109 (CHAES)
Benchmarks: DIHARD3-eval CALLHOME-part2 CH109 AMI test LibriSpeechMix
Metrics: DER WER cpWER
Baselines: MSDD EEND-EDA WavLM-L+EEND-VC EEND-GLA-Large AED-EEND Canary-170M SOT-ASR SOT-ASR-SQR DOM-SOT MT-LLM MS-Canary
Builds on: EEND-EDA Canary Sequence-to-sequence serialized output training (SOT) NEST Permutation invariant training
Closest papers by meaning
- Zipformer: A faster and better encoder for automatic speech recognition (ICLR 2024 · similarity 0.83)
- Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations (AAAI 2024 · similarity 0.82)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.82)
- FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing (NeurIPS 2025 · similarity 0.81)
- Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation (NeurIPS 2024 · similarity 0.81)
- PAT: Pruning-Aware Tuning for Large Language Models (AAAI 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.