Research map / Image generation / Editing and style transfer
Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes two zero-shot audio editing methods built on edit-friendly DDPM inversion with AudioLDM2: ZETA (text-prompt-based editing) and ZEUS (unsupervised editing via posterior principal components in the denoiser).
Problem
Zero-shot editing of audio signals (especially music) with pre-trained diffusion models has not been explored, and existing audio editing methods either need training on edit triplets or test-time optimization; text-based editing also cannot express many musical variations.
Approach
Adapts DDPM-inversion-based text editing from images to audio (ZETA), and introduces ZEUS, which finds semantic editing directions as top posterior principal components of the denoiser's Jacobian (posterior covariance) in an unsupervised way and adds them during DDPM sampling.
Key results
- ZETA achieves CLAP ~0.34 comparable to baselines while LPAPS values are 4.29 vs 4.87 (SDEdit), 6.26 (DDIM inversion), 5.74 (MusicGen) on one example, indicating better structure preservation.
- In an AMT user study with 277 valid participants, ZETA was clearly preferred over DDIM inversion, MusicGen, and SDEdit for matching the text edit while keeping the original essence.
- ZEUS achieves lower FAD (better musical quality) than SDEdit for any given edit strength, measured against MusicDelta and FMA-pop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZETA instead of SDEdit for text-based editing | AudioLDM2 · MusicDelta (MedleyDB), text-based editing | LPAPS: 4.87 -> 4.29 | Helped |
| Use ZETA instead of DDIM Inversion for text-based editing | AudioLDM2 · MusicDelta (MedleyDB), text-based editing | LPAPS: 6.26 -> 4.29 | Helped |
| Use ZETA instead of MusicGen for text-based editing | AudioLDM2 · MusicDelta (MedleyDB), text-based editing | LPAPS: 5.74 -> 4.29 | Helped |
| Lower target-prompt CFG strength below 12 (ablation) | ZETA · MusicDelta (MedleyDB), text-based editing | LPAPS at matched CLAP | Helped |
| Vary source-prompt CFG strength (ablation) | ZETA · MusicDelta (MedleyDB), text-based editing | CLAP and LPAPS | No clear effect |
| Use ZETA rather than SDEdit at matched edit strength | AudioLDM2 unsupervised editing · MusicDelta (MedleyDB), unsupervised editing | FAD to FMA-pop | Helped |
Surprising findings
- Partial DDIM inversion (stopped mid-way) achieves relatively good CLAP-LPAPS results, though it lacks text adherence compared to other methods.
Limitations
- Edit quality is bounded by the pre-trained audio model and the target prompt's expressiveness.
- Unsupervised PCs are not controllable; a user cannot know in advance which PC yields a desired edit.
- Data leakage through the UNet/attention architecture can spread localized edits across the signal, requiring masking heuristics.
Experimental setup
Datasets: MedleyDB (MusicDelta subset, 34 excerpts) AudioSet FMA-pop MedleyMDPrompts (new prompts, 107 source and 696 target prompts)
Benchmarks: MedleyDB MusicDelta with MedleyMDPrompts
Metrics: CLAP score LPAPS FAD (CLAP features) user preference rate
Baselines: MusicGen (melody-conditioned) SDEdit DDIM inversion Partial DDIM inversion
Builds on: DDPM inversion (edit-friendly) SDEdit DDIM inversion AudioLDM2 Posterior covariance via Jacobian of MSE denoiser (Kadkhodaie/Stanczuk et al.)
Closest papers by meaning
- Prompt-guided Precise Audio Editing with Diffusion Models (ICML 2024 · similarity 0.87)
- SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor (AAAI 2025 · similarity 0.86)
- Music Style Transfer with Time-Varying Inversion of Diffusion Models (AAAI 2024 · similarity 0.86)
- ZeroSep: Separate Anything in Audio with Zero Training (NeurIPS 2025 · similarity 0.86)
- SAO-Instruct: Free-form Audio Editing using Natural Language Instructions (NeurIPS 2025 · similarity 0.85)
- DITTO: Diffusion Inference-Time T-Optimization for Music Generation (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.