Research map / Video / Video generation
Tell What You Hear From What You See - Video to Audio Generation Through Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VATT, an LLM-based video-to-audio generator that maps video to audio captions and uses a masked-token bidirectional transformer over Encodec tokens to generate audio, optionally steered by text prompts.
Problem
Video-to-audio generation lacks controllability and context sensitivity; existing methods condition only on visual frames and cannot be steered by text or suggest audio descriptions for a video.
Approach
Fine-tune an instruction-tuned LLM (LoRA plus a linear video projector) to produce audio captions from video, then feed its last-layer hidden states along with video and optional text prompts into a bidirectional masked-token transformer that predicts Encodec audio tokens with iterative parallel decoding.
Key results
- VATT-LLama-T with GT audio captions reaches KLD 1.41 on VGGSound, versus 2.09 for the best text-to-audio baseline AudioLDM-2 (GT caption).
- Without text prompt, VATT-Gemma achieves KLD 2.25 and Align Acc 82.81, the best among video-to-audio methods, with 0.65 s generation time per sample versus 4.4 s for Diff-Foley.
- In video-to-audio captioning, VATT Converter-LLaMA reaches CLAP score 0.263 versus 0.213 for LLaVA with visual prompt, reported as a +5.0% relative accuracy gain in audio-text relevance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GT audio captions as prompts | VATT-LLama · VGGSound test | KLD: 2.39 -> 1.41 | Helped |
| Use GT audio captions as prompts | VATT-Gemma · VGGSound test | KLD: 2.25 -> 1.66 | Helped |
| Add the LLM converter to audio-relevant visual conditioning (ablation) | VATT-Gemma · VGGSound test | KLD: 2.43 -> 2.25 | Helped |
| Use bidirectional self-attention instead of interleaving cross-attention (ablation) | VATT-Gemma · VGGSound test | KLD: 2.76 -> 2.25 | Helped |
| Use uniform rather than the selected Gaussian masking-ratio distribution (ablation) | VATT-Gemma · VGGSound test | KLD: 2.25 -> 2.52 | Hurt |
| Use arc cosine rather than Gaussian mean 0.75 masking (ablation) | VATT-Gemma · VGGSound test | KLD: 2.25 -> 2.34 | No clear effect |
| Set Gaussian masking-ratio mean to 0.55 instead of 0.75 (ablation) | VATT-Gemma · VGGSound test | KLD: 2.25 -> 2.31 | Hurt |
| Set Gaussian masking-ratio mean to 0.95 instead of 0.75 (ablation) | VATT-Gemma · VGGSound test | FAD: 2.36 -> 2.49 | Mixed |
| Use a moving Gaussian masking-ratio mean instead of fixed mean 0.75 (ablation) | VATT-Gemma · VGGSound test | KLD: 2.25 -> 2.42 | Hurt |
| Use a self-generated LLama caption prompt instead of no prompt (ablation) | VATT-LLama · VGGSound test | KLD: 2.39 -> 2.38 | No clear effect |
| Use a Gemma-generated caption prompt instead of no prompt (ablation) | VATT-Gemma · VGGSound test | FAD: 2.35 -> 3.67 | Hurt |
| Use GT audio captions rather than self-generated captions (ablation) | VATT-LLama-T · VGGSound test | KLD: 2.38 -> 1.41 | Helped |
Surprising findings
- Feeding the model its own generated captions performs worse than no prompt, and the GT-caption setting gives the largest gain, suggesting caption quality is the bottleneck.
- V2A-Mapper achieves the best FAD (0.99) on VGGSound, beating the proposed method, which the authors attribute to its AudioLDM-based diffusion backbone.
Limitations
- Text diversity from current audio LLMs is limited, so user prompts with differing styles can conflict with audio quality and instruction adherence.
- Fidelity is weaker than diffusion-based V2A-Mapper; token-based models are less competitive on audio quality.
- Synthetic captions come from LTU-13B, and generated captions used as self-prompts degrade results relative to no prompt.
Experimental setup
Datasets: VGGSound AudioSet-2M AudioCaps V2A Instruction
Benchmarks: VGGSound test AudioCaps test
Metrics: KLD FAD Align Acc CLAP score Speed BERTScore BLEU-4 ROUGE-L CIDEr
Baselines: SpecVQGAN IM2WAV Diff-Foley FoleyGen V2A-Mapper AudioGen AudioLDM-2 LLaVA-13B-v1.5 Video-LLaMA-7B
Builds on: MaskGIT SoundStorm Encodec VampNet Video-LLaMA
Closest papers by meaning
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.90)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.90)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.88)
- AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (ICML 2026 · similarity 0.88)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.88)
- Audio-Sync Video Generation with Multi-Stream Temporal Control (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.