Research map / Video / Video generation
ViSAGe: Video-to-Spatial Audio Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the task of generating first-order ambisonics directly from silent field-of-view videos, with a new YT-Ambigen dataset, spatial energy-map metrics, and ViSAGe, an autoregressive DAC-codec transformer with a new code generation pattern and directional/visual guidance.
Problem
Generating spatial audio (first-order ambisonics) directly from silent videos, which existing video-to-audio models (mono only) and audio spatialization models (require reference mono audio) cannot do end-to-end.
Approach
An end-to-end transformer encoder-decoder that conditions on CLIP features, patchwise energy maps, and a camera direction embedding, and autoregressively generates all four FOA channels' DAC codes using a 2L+1-step pattern that models both residual and spatial dependencies, with classifier-free guidance on direction and visual conditions plus azimuth rotation augmentation.
Key results
- ViSAGe (Directional) achieves CC_all 0.721 and AUC_all 0.890 versus best two-stage baseline CC_all 0.619 and AUC_all 0.848 (SpecVQGAN + audio spatialization).
- ViSAGe (Directional & Visual) reaches FAD_dec 3.86 and KLD_dec 1.71, improving over the best two-stage semantic scores (FAD_dec 5.68 for Diff-Foley + ambisonics encoding).
- In a 12-participant subjective test, ViSAGe was preferred overall over SpecVQGAN (50.00% win vs 23.33% lose) and Diff-Foley (44.44% win vs 16.67% lose).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ViSAGe with directional guidance instead of the two-stage baselines | Video-to-spatial audio generation · YT-Ambigen test | CC_all: 0.577 -> 0.721 | Helped |
| add visual guidance alongside directional guidance (ablation) | ViSAGe · YT-Ambigen test | FAD_dec: 5.56 -> 3.86 | Helped |
| add visual guidance alongside directional guidance (ablation) | ViSAGe · YT-Ambigen test | CC_all: 0.721 -> 0.635 | Hurt |
| add direction embedding (ablation) | ViSAGe · YT-Ambigen test | CC_all: 0.430 -> 0.524 | Helped |
| add patchwise energy maps (ablation) | ViSAGe · YT-Ambigen test | CC_all: 0.524 -> 0.531 | Helped |
| add rotation augmentation (ablation) | ViSAGe · YT-Ambigen test | CC_all: 0.531 -> 0.635 | Helped |
| pretrain on VGGSound (ablation) | ViSAGe · YT-Ambigen test | FAD_dec: 3.74 -> 3.86 | No clear effect |
| use the proposed code generation pattern instead of Sequential Delay (ablation) | ViSAGe · YT-Ambigen test | FAD_dec: 10.75 -> 4.44 | Helped |
| model both residual and spatial dependencies rather than residual dependency only (ablation) | ViSAGe · YT-Ambigen test | CC_all: 0.480 -> 0.544 | Helped |
| model both residual and spatial dependencies rather than spatial dependency only (ablation) | ViSAGe · YT-Ambigen test | FAD_dec: 5.89 -> 4.44 | Helped |
| add directional guidance (ablation) | ViSAGe classifier-free guidance · YT-Ambigen test | CC_all: 0.488 -> 0.635 | Helped |
| add visual guidance (ablation) | ViSAGe classifier-free guidance · YT-Ambigen test | FAD_dec: 5.56 -> 3.86 | Helped |
Surprising findings
- Manually encoding the generated W-channel into FOA gives better semantic quality than the learned audio spatialization model, but poorer spatial accuracy.
- The sequential delay pattern from stereo music generation fails badly for FOA, with FAD_dec 10.75 versus 4.44 for the proposed pattern.
Limitations
- Relies on field-of-view videos with a camera direction input rather than full panoramic scene context.
- Guidance with visual condition slightly degrades spatial accuracy relative to directional-only guidance.
- Evaluated on 5-second clips only; subjective test uses only 12 participants with 30 samples.
Experimental setup
Datasets: VGGSound YT-Ambigen YT360
Benchmarks: YT-Ambigen
Metrics: FAD_dec KLD_dec FAD_avg CC AUC subjective preference
Baselines: SpecVQGAN + ambisonics encoding SpecVQGAN + audio spatialization Diff-Foley + ambisonics encoding Diff-Foley + audio spatialization Sequential delay pattern Residual only pattern Spatial only pattern
Builds on: Descript Audio Codec SpecVQGAN Diff-Foley FoleyGen PAVER
Closest papers by meaning
- OmniAudio: Generating Spatial Audio from 360-Degree Video (ICML 2025 · similarity 0.89)
- Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation (ICLR 2025 · similarity 0.87)
- MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations (NeurIPS 2025 · similarity 0.86)
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.85)
- SonicVisionLM: Playing Sound with Vision Language Models (CVPR 2024 · similarity 0.85)
- 🎧MOSPA: Human Motion Generation Driven by Spatial Audio (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.