Research map / Label-efficient and robust learning / Recognition and long-tail
Weakly-supervised Audio Separation via Bi-modal Semantic Similarity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a weakly-supervised language-conditional audio separation framework that uses frozen CLAP embeddings as a contrastive signal on separated sources, combined with mix-and-separate losses, to train without single-source audio.
Problem
Conditional sound separation normally needs single-source audio for supervision; mix-and-separate training without single-source examples suffers a train/test distribution shift and degrades sharply as the number of sources in training mixtures grows.
Approach
Use single-source text entities extracted from captions and a frozen pretrained CLAP audio-text encoder pair to supply a contrastive weak-supervision loss on predicted single sources, combined with a mixture-of-mixtures reconstruction loss and a consistency reconstruction loss; a residual conditional U-Net with multi-scale cross-attention conditioning is used as the separator.
Key results
- In the unsupervised 2-source setting on MUSIC, the proposed method reaches 7.9 SDR versus 8.1 for the supervised CLAPSep upper bound (about 97.5% of supervised performance) and 4.6 for CLIPSep-Text.
- In the semi-supervised setting with 5% single-source MUSIC data plus 95% mixtures, SDR reaches 8.8, beating the fully supervised baseline at 8.1.
- On AudioCaps natural mixtures, semi-supervised training using VGGSound as supervised data reaches 4.3 SDR versus 2.9 for the unsupervised weakly-supervised baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Bi-modal CLAP weak supervision instead of mix-and-separate baselines | MUSIC audio separation · MUSIC 2-source test; 2-source training | SDR: +4.7 SDR | Helped |
| Use Bi-modal CLAP weak supervision instead of image-conditional baselines | MUSIC audio separation · MUSIC 2-source test; 2-source training | SDR: +3.8, +3.0, and +3.3 SDR | Helped |
| Use Bi-modal CLAP weak supervision rather than the pretrained timbre classifier (ablation) | MUSIC audio separation · MUSIC 2-source test; 2-, 3-, and 4-source training | SDR: 7.9 $\pm$ 0.35 vs. 6.1 $\pm$ 0.33; 7.1 $\pm$ 0.42 vs. 5.2 $\pm$ 0.37; 6.2 $\pm$ 0.38 vs. 4.1 $\pm$ 0.35 | Helped |
| Use a pretrained timbre classifier rather than training it concurrently (ablation) | Proposed weakly supervised MUSIC separation · MUSIC 2-source test; 2-, 3-, and 4-source training | SDR: 5.0 $\pm$ 0.29 -> 6.1 $\pm$ 0.33; 4.5 $\pm$ 0.32 -> 5.2 $\pm$ 0.37; 3.5 $\pm$ 0.27 -> 4.1 $\pm$ 0.35 | Helped |
| Add 95% unsupervised mixture data to 5% supervised MUSIC data | MUSIC supervised audio separation · MUSIC-2Mix | SDR: 2.6 $\pm$ 0.33 -> 8.8 $\pm$ 0.28 | Helped |
| Add weakly supervised mixture training with 5% supervised data instead of using all data as supervised | MUSIC supervised audio separation · MUSIC-2Mix | SDR: 8.1 $\pm$ 0.31 -> 8.8 $\pm$ 0.28; +0.7 | Helped |
| Add 95% unsupervised mixture data to 5% supervised VGGSound data | VGGSound supervised audio separation · VGGSound-2Mix | SDR: 2.3 $\pm$ 0.23 -> 3.1 $\pm$ 0.31 | Helped |
| Add supervised VGGSound data to unsupervised AudioCaps training | AudioCaps separation · AudioCaps-2Mix | SDR: 2.9 $\pm$ 0.23 -> 4.3 $\pm$ 0.34; +1.4 | Helped |
Surprising findings
- Replacing the CLAP-based loss with a timbre classifier loss performs worse than several baselines that use no weak supervision when the classifier is trained concurrently.
- Semi-supervised training with only 5% of single-source MUSIC data outperforms the supervised baseline trained on 100% of it.
Limitations
- Relies on a pretrained joint audio-language embedding model (CLAP) and on extracting single-source entities from captions.
- Performance still lags supervised training on some settings and the multi-source baselines degrade with more sources.
- Evaluated mainly on synthetic mixtures of 2 sources; natural mixture evaluation is limited to synthetic MoM built from AudioCaps test set.
Experimental setup
Datasets: MUSIC VGGSound AudioCaps
Benchmarks: MUSIC-2Mix VGGSound-2Mix AudioCaps-2Mix
Metrics: SDR SIR SAR
Baselines: PIT MixIT MixPIT CLIPSep-Img CoSep SOP CLIPSep-Text BertSep CLAPSep LASS-Net Weak-Sup
Builds on: mix-and-separate CLIPSep CLAP mixture invariant training MixPIT
Closest papers by meaning
- OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup (ICLR 2025 · similarity 0.87)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.87)
- SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes (ICLR 2025 · similarity 0.86)
- ZeroSep: Separate Anything in Audio with Zero Training (NeurIPS 2025 · similarity 0.86)
- ADIFF: Explaining audio difference using natural language (ICLR 2025 · similarity 0.86)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.