Research map / Image generation / Autoregressive and latent generation
ZeroSep: Separate Anything in Audio with Zero Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a pre-trained text-guided audio diffusion model (AudioLDM/AudioLDM2/Tango) can perform zero-shot text-queried source separation via DDIM/DDPM latent inversion of the mixture followed by text-conditioned denoising with classifier-free guidance weight 1.
Problem
Audio source separation normally requires large task-specific supervised training data and struggles with open-set real-world acoustic scenes; the paper asks whether pre-trained text-guided audio diffusion models can separate sources without any training.
Approach
Invert the mixed audio into a diffusion latent space (DDIM/DDPM inversion with a null or other-source prompt), then denoise with a text prompt describing the target source and classifier-free guidance weight omega=1, which removes the unconditional term and isolates the target; omega>1 causes generation rather than separation.
Key results
- On MUSIC, ZeroSep reaches FAD 0.377, LPAPS 4.669, CLAP-A 0.615 and CLAP-T 0.271, outperforming FlowSep (FAD 0.402, CLAP-A 0.564) across metrics.
- On AVE, ZeroSep achieves FAD 0.269 and LPAPS 4.537, comparable to FlowSep (FAD 0.258, CLAP-A 0.493 vs ZeroSep 0.442), but with much lower CLAP-T (-0.001 vs 0.082 for FlowSep).
- Separation quality correlates positively with generation quality across AudioLDM variants; guidance weight omega=1 is best, and omega>1 degrades performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ZeroSep instead of FlowSep | MUSIC separation · MUSIC test | FAD: 0.402 -> 0.377 | Helped |
| use ZeroSep instead of FlowSep | MUSIC separation · MUSIC test | C-A: 0.564 -> 0.615 | Helped |
| use ZeroSep instead of FlowSep | AVE separation · AVE test | FAD: 0.258 -> 0.269 | Hurt |
| use ZeroSep instead of FlowSep | AVE separation · AVE test | LPAPS: 4.719 -> 4.537 | Helped |
| replace the target reverse prompt with a random prompt (ablation) | ZeroSep on MUSIC · MUSIC test | C-A: 0.615 -> 0.363 | Hurt |
| replace the target reverse prompt with a random prompt (ablation) | ZeroSep on AVE · AVE test | FAD: 0.269 -> 0.325 | Hurt |
| replace the null inversion prompt with a non-target source prompt (ablation) | ZeroSep on MUSIC · MUSIC test | LPAPS: 4.669 -> 4.547 | Helped |
| replace the null inversion prompt with a non-target source prompt (ablation) | ZeroSep on MUSIC · MUSIC test | FAD: 0.377 -> 0.454 | Hurt |
| replace the null inversion prompt with a non-target source prompt (ablation) | ZeroSep on AVE · AVE test | C-A: 0.442 -> 0.496 | Helped |
| replace the null inversion prompt with a non-target source prompt (ablation) | ZeroSep on AVE · AVE test | FAD: 0.269 -> 0.321 | Hurt |
| increase classifier-free guidance from 0 to 1 (ablation) | ZeroSep · separation evaluation | LPAPS | Helped |
| increase classifier-free guidance above 1 (ablation) | ZeroSep · separation evaluation | LPAPS | Hurt |
Surprising findings
- Traditional SDR/SIR/SAR metrics are misleading for generative separation: a VAE+vocoder reconstruction of the mixture scores very poorly (SDR -23.07) despite being perceptually faithful.
- Classifier-free guidance weight above 1, the standard setting for generation, degrades separation below the unguided mixture baseline.
Limitations
- Only tested on AudioLDM family and Tango; larger or alternative architectures untested due to computational constraints.
- Reliance on latent inversion introduces approximation errors that can impair separation fidelity.
- Failure when an interfering source has significant energy that is not removed in a single denoising pass; may need more steps or iterative application.
Experimental setup
Datasets: AVE MUSIC AudioCap
Benchmarks: MUSIC AVE
Metrics: FAD LPAPS CLAP-A CLAP-T SDR SIR SAR
Baselines: LASS-Net AudioSep FlowSep NMF-MFCC AudioEdit
Builds on: DDIM DDPM inversion AudioLDM AudioLDM2 Tango
Closest papers by meaning
- Multi-Source Diffusion Models for Simultaneous Music Generation and Separation (ICLR 2024 · similarity 0.87)
- Weakly-supervised Audio Separation via Bi-modal Semantic Similarity (ICLR 2024 · similarity 0.86)
- Prompt-guided Precise Audio Editing with Diffusion Models (ICML 2024 · similarity 0.86)
- ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Prior (ICML 2025 · similarity 0.86)
- Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion (ICML 2024 · similarity 0.86)
- MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.