Research map / Large language models / Language, safety and interpretability
SAO-Instruct: Free-form Audio Editing using Natural Language Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Stable Audio Open on 150k synthetic and manually edited audio editing triplets so it can edit audio from free-form natural-language instructions alone.
Problem
Editing existing audio with free-form natural language instructions is underexplored; prior zero-shot methods need full target descriptions and instruction-based methods (AUDIT) support only predefined edit operations.
Approach
Build a dataset of (input audio, edit instruction, output audio) triplets from three sources (Prompt-to-Prompt, DDPM inversion with Bayesian-optimized parameters, and deterministic manual edits with LLM-written instructions) and fine-tune Stable Audio Open to take the input audio latent plus instruction as conditioning.
Key results
- SAO-Instruct achieves the best subjective relevance (3.83) and faithfulness (3.99) MOS against ZETA and AudioEditor, with quality 3.54 close to ZETA's 3.56.
- On deterministic manual edit tasks, SAO-Instruct reduces average MR-STFT to 2.82 versus 4.65 for the best ZETA variant.
- Inference takes 9.94 s per sample, roughly 8x faster than AudioEditor (79.49 s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on combined dataset (P2P + DDPM inversion + manual edits) instead of a single generation method (ablation) | SAO-Instruct · AudioCaps test (1k 10s samples) | FD / KL (regenerated ref) and CLAP: Combined-Large CLAP 0.38 vs 0.34-0.35 for DDPM/manual; FD orig 18.38 vs 14.60 for manual-only | Mixed |
| train only on Prompt-to-Prompt data (fully synthetic) (ablation) | SAO-Instruct · AudioCaps test (1k 10s samples) | FD regenerated ref: 18.29 (best regenerated FD) but LSD/KL to original worse | Helped |
| train only on manual edit data (ablation) | SAO-Instruct · AudioCaps test (1k 10s samples) | FD original ref: 14.60 (best original-ref FD and KL 0.58) | Helped |
| sample initial latent from noised input-audio latent instead of pure noise (ablation) | SAO-Instruct inference · AudioCaps test (1k 10s samples) | FD original ref and CLAP: FD 18.38 vs 29.73; CLAP 0.38 vs 0.32 | Helped |
| use SAO-Instruct (free-form instruction only) instead of ZETA/AudioEditor with full captions | Editing vs zero-shot baselines · Subjective listening study (AudioCaps, 13 participants) | Relevance and Faithfulness MOS: Relevance 3.83 vs 3.33 (AudioEditor); Faithfulness 3.99 vs 2.95 (ZETA T=50) | Helped |
| use SAO-Instruct instead of AudioEditor | Inference speed · AudioCaps test inference time | Inference time per sample: 9.94 s vs 79.49 s | Helped |
| use SAO-Instruct instead of ZETA/AudioEditor baselines | Deterministic manual editing tasks · Average across 10 deterministic tasks | STFT and SI-SDR: STFT 2.91 vs 4.73 (best ZETA); SI-SDR -21.62 vs -28.75 | Helped |
| use SAO-Instruct instead of ZETA baseline | Objective reference-based metrics (FD/LSD/KL) · AudioCaps test | FD regenerated ref / KL regenerated ref: FD regen 18.97 vs 17.01 (ZETA T=50); KL regen 1.76 vs 1.26 | Hurt |
| use SAO-Instruct instead of AudioEditor and ZETA on Production Quality | AudioBox Aesthetics PQ · AudioCaps test | Production Quality: 5.61 vs 6.06 (ZETA T=50) | Hurt |
| use SAO-Instruct instead of baselines on Production Complexity | AudioBox Aesthetics PC · AudioCaps test | Production Complexity: 3.23 vs 3.02 (AudioEditor) | Helped |
| use SAO-Instruct instead of AudioEditor on CLAP score and IS | CLAP and Inception Score · AudioCaps test | CLAP / IS: CLAP 0.38 vs 0.48 (AudioEditor); IS 7.59 vs 10.01 | Hurt |
Surprising findings
- AudioEditor has the highest CLAP and IS scores yet lower faithfulness, suggesting more aggressive edits.
- Prompt-to-Prompt-only training yields the highest regenerated-reference similarity but lower faithfulness to the original audio, while manual-edit training scores worst on regenerated FD.
Limitations
- Prompt-to-Prompt and DDPM inversion data generation is computationally expensive and bounded by the base generator's capabilities.
- Objective metrics FD/LSD/KL reward similarity to references and do not reliably indicate edit success; SAO-Instruct slightly underperforms ZETA on production quality.
- Only English instructions and single-step edits are supported.
Experimental setup
Datasets: AudioCaps WavCaps AudioSet-SL FreeSound MACS FMA
Benchmarks: AudioCaps test subset
Metrics: FD LSD KL IS CLAP MOS quality MOS relevance MOS faithfulness STFT loss MR-STFT
Baselines: ZETA AudioEditor
Builds on: Stable Audio Open Prompt-to-Prompt DDPM inversion ZETA InstructPix2Pix AUDIT
Closest papers by meaning
- InstructSpeech: Following Speech Editing Instructions via Large Language Models (ICML 2024 · similarity 0.90)
- Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts (NeurIPS 2026 · similarity 0.88)
- AVENUE: Audio-Video Editing Understanding and Evaluation (NeurIPS 2026 · similarity 0.87)
- Prompt-guided Precise Audio Editing with Diffusion Models (ICML 2024 · similarity 0.87)
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.87)
- ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.