Research map / 3D vision / 3D generation
SoundBrush: Sound as a Brush for Visual Scene Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SoundBrush, which maps CLAP audio features into text-token space of an InstructPix2Pix-based latent diffusion editor to edit images and 3D scenes with sound, trained on an 83K sound-paired synthetic and real-inpainted editing dataset.
Problem
Editing existing visual scenes guided by in-the-wild sound while preserving original content; prior sound-guided methods either generate new scenes, only change style, or alter source structure, and paired sound-image editing data is scarce.
Approach
Translate CLAP audio features through a Transformer mapping network into CLIP text-space tokens that replace text instructions in InstructPix2Pix, jointly trained with LoRA on an automatically constructed sound-paired before/after editing dataset, plus an InfoNCE alignment loss and l1 regularization.
Key results
- SoundBrush achieves AVS 0.261, IIS 0.772, TVS 0.203, FID 131.7 in its final configuration, best on AVS, IIS and TVS in the ablation.
- InstructAny2Pix attains better FID than SoundBrush in the quantitative comparison but degrades on other metrics, per the authors.
- In a 47-participant MOS user study over 20 samples, SoundBrush is preferred on all three criteria, with the largest gap on Q3 (edit with sound while keeping structure).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the audio-token count from 1 to 5 (ablation) | SoundBrush · SoundBrush ablation evaluation | AVS: 0.235 -> 0.261 | Helped |
| increase the audio-token count from 1 to 5 (ablation) | SoundBrush · SoundBrush ablation evaluation | IIS: 0.761 -> 0.772 | Helped |
| increase the audio-token count from 1 to 5 (ablation) | SoundBrush · SoundBrush ablation evaluation | TVS: 0.192 -> 0.203 | Helped |
| increase the audio-token count from 1 to 5 (ablation) | SoundBrush · SoundBrush ablation evaluation | FID: 131.8 -> 131.7 | Helped |
| add L_NCE (ablation) | SoundBrush with 5 audio tokens · SoundBrush ablation evaluation | AVS: 0.258 -> 0.261 | Helped |
| add L_NCE (ablation) | SoundBrush with 5 audio tokens · SoundBrush ablation evaluation | IIS: 0.767 -> 0.772 | Helped |
| add L_NCE (ablation) | SoundBrush with 5 audio tokens · SoundBrush ablation evaluation | TVS: 0.199 -> 0.203 | Helped |
| add L_NCE (ablation) | SoundBrush with 5 audio tokens · SoundBrush ablation evaluation | FID: 138.3 -> 131.7 | Helped |
| increase the audio-token count from 5 to 10 (ablation) | SoundBrush with L_NCE · SoundBrush ablation evaluation | AVS: 0.261 -> 0.233 | Hurt |
| increase the audio-token count from 5 to 10 (ablation) | SoundBrush with L_NCE · SoundBrush ablation evaluation | IIS: 0.772 -> 0.763 | Hurt |
| increase the audio-token count from 5 to 10 (ablation) | SoundBrush with L_NCE · SoundBrush ablation evaluation | TVS: 0.203 -> 0.191 | Hurt |
| increase the audio-token count from 5 to 10 (ablation) | SoundBrush with L_NCE · SoundBrush ablation evaluation | FID: 131.7 -> 136.1 | Hurt |
Surprising findings
- Increasing audio tokens from five to ten degraded performance, and a single token was insufficient.
- InstructAny2Pix obtains better FID than SoundBrush despite worse editing-related metrics.
Limitations
- Sometimes inserts objects without proper spatial understanding (e.g., train in the sky, wrong object scale).
- Images with human-related sound sources often show low quality, attributed to low-resolution, motion-blurred video training data.
- Does not specify where sounding objects are inserted; relies only on audio input.
Experimental setup
Datasets: VGGSound UrbanSound8K
Benchmarks: VGGSound
Metrics: AVS (ImageBind) IIS (CLIP) TVS (CLIP) FID MOS user study
Baselines: AudioToken GlueGen InstructAny2Pix SonicDiffusion
Builds on: InstructPix2Pix Latent Diffusion Model Prompt-to-Prompt InstructNeRF2NeRF CLAP
Closest papers by meaning
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.88)
- SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding (CVPR 2025 · similarity 0.88)
- Sounding that Object: Interactive Object-Aware Image to Audio Generation (ICML 2025 · similarity 0.88)
- NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields (ICLR 2025 · similarity 0.87)
- Learning to Highlight Audio by Watching Movies (CVPR 2025 · similarity 0.86)
- Prompt-guided Precise Audio Editing with Diffusion Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.