Research map / Image generation / Editing and style transfer
Prompt-guided Precise Audio Editing with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PPAE, a training-free precise audio editing method that manipulates cross-attention maps of a pretrained text-to-audio diffusion model (Tango) with a cosine fuser and guidance bootstrapping.
Problem
Precise text-guided audio editing: modifying target sound events within an audio track while leaving unrelated events and overall structure unchanged, without requiring masks or editing-specific training.
Approach
Injects cross-attention maps from the source prompt into the target-prompt denoising process of a pretrained latent diffusion TTA model, with a cosine-annealed Fuser to blend attention smoothly, a DDPM-based edit-friendly inversion, and a CLAP-filtered guidance-scale bootstrapping over a list of scales.
Key results
- Replace task: PPAE achieves FD 27.53 and KL 1.30 vs. PTP 45.91 and 4.42 and Regenerated 32.94 and 1.69.
- Refine task: PPAE achieves FAD 6.86 and CLAP 0.63 vs. unedited 8.19 and 0.25.
- Subjective evaluation (14 participants): PPAE consistency 95.0 vs. regenerated 81.42 on Replace.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace audio event using PPAE instead of regenerating the full audio | Tango · Audio Replace | FD: 32.94 -> 27.53 | Helped |
| Replace audio event using PPAE instead of regenerating the full audio | Tango · Audio Replace | CLAP: 0.63 -> 0.62 | No clear effect |
| Refine audio with PPAE rather than leaving it unedited | Tango · Audio Refine | CLAP: 0.25 -> 0.63 | Helped |
| Increase reweight control from c=0 to c=2 | PPAE · Audio Reweight: target event | CLAP: 0.35 -> 0.83 | Helped |
| Increase reweight control from c=0 to c=-1 | PPAE · Audio Reweight: target event | CLAP: 0.35 -> 0.10 | Hurt |
| Set guidance scale w=25 rather than w=7 (ablation) | PPAE generation configuration · Audio editing configuration ablation | FAD: 5.97 -> 3.34 | Helped |
| Set guidance scale w=75 rather than w=25 (ablation) | PPAE generation configuration · Audio editing configuration ablation | CLAP: 0.68 -> 0.69 | Helped |
| Add self-replace steps (0.2) (ablation) | PPAE generation configuration · Audio editing configuration ablation | FAD: 3.67 -> 12.92 | Hurt |
| Use CosineAnnealing instead of a Fuser-free PTP setup (ablation) | PPAE · Audio editing Fuser ablation | FAD: 5.36 -> 3.15 | Helped |
| Use CosineAnnealing rather than Linear scheduling (ablation) | PPAE Fuser · Audio editing Fuser ablation | FAD: 3.47 -> 3.15 | Helped |
| Remove guidance bootstrapping (ablation) | PPAE · Audio editing bootstrapping ablation | FAD: 3.38 -> 5.52 | Hurt |
| Increase CLAP-selected candidates from 1 to 10 (ablation) | PPAE bootstrapping · Audio editing bootstrapping ablation | CLAP: 0.63 -> 0.70 | Helped |
Surprising findings
- Regenerating audio from the target prompt fails on consistency, showing regeneration alone does not preserve unrelated content.
- Attention-map fusion without a scheduler (plain PTP) yields the most drastic changes and worse metrics than the cosine Fuser.
Limitations
- Relies on accurate inversion; audio outside the model's training domain is hard to edit precisely.
- Attention-level edits restrict the extent of changes and are unsuitable for large structural alterations.
- Diffusion sampling is slow, limiting real-time use.
Experimental setup
Datasets: FSD50K
Benchmarks: FSD50K-based editing test set (100 per task)
Metrics: FAD LSD FD KL CLAP score subjective relevance subjective consistency
Baselines: Prompt-to-Prompt (reimplemented for audio) Regenerated audio Audit Unedited audio
Builds on: Prompt-to-Prompt Null-text Inversion Tango Edit-friendly inversion
Closest papers by meaning
- SAO-Instruct: Free-form Audio Editing using Natural Language Instructions (NeurIPS 2025 · similarity 0.87)
- Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion (ICML 2024 · similarity 0.87)
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.86)
- ZeroSep: Separate Anything in Audio with Zero Training (NeurIPS 2025 · similarity 0.86)
- SoundBrush: Sound as a Brush for Visual Scene Editing (AAAI 2025 · similarity 0.86)
- AVENUE: Audio-Video Editing Understanding and Evaluation (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.