Research map / Video / Video generation
VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VinTAGe, a flow-based transformer that jointly conditions audio generation on video and text, guided by pretrained T2A and V2A teacher models, and introduces the VinTAGe-Bench benchmark of 636 video-text-audio pairs for holistic (onscreen+offscreen) audio generation.
Problem
Text-to-audio and video-to-audio models cannot jointly produce holistic audio: T2A ignores on-screen alignment and V2A misses off-screen sounds. Existing joint text-video methods also show modality bias favoring video.
Approach
A SiT-based flow-matching transformer with a Visual-Text encoder (gated cross-attention, optical-flow and frame-index temporal cues) and learnable per-layer weighting of text vs. visual cross-attention, trained with teacher-guided alignment losses from frozen-style pretrained text-only and video-only audio generators to reduce modality bias.
Key results
- On VinTAGe-Bench VinTAGe reports FAD 3.05 and FID 16.43, the best on both, with mean alignment 16.06 and mean concept accuracy 50.66%.
- On VGGSound VinTAGe achieves FID 6.65 and MKL 4.12, outperforming FoleyCrafter (9.17 FID) and Diff-Foley (9.87 FID) while running at 4 FPS.
- Ablation shows removing the guidance loss raises FID from 16.43 to 19.48 and lowers mean concept accuracy from 50.66% to 43.15%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add video captions to Tango2 | Tango2 · VinTAGe-Bench | FAD: 5.85 -> 4.12 | Helped |
| add teacher guidance loss (ablation) | VinTAGe · VinTAGe-Bench | FID: 19.48 -> 16.43 | Helped |
| add cross-attention in the VT-Encoder (ablation) | VinTAGe · VinTAGe-Bench | Mean-Acc (%): 41.66 -> 50.66 | Helped |
| add augmentation (ablation) | VinTAGe · VinTAGe-Bench | Mean-Acc (%): 38.55 -> 50.66 | Helped |
| add mean optical-flow feature (ablation) | VinTAGe · VinTAGe-Bench | FID: 26.36 -> 16.43 | Helped |
| add frame-index feature (ablation) | VinTAGe · VinTAGe-Bench | FID: 25.94 -> 16.43 | Helped |
Surprising findings
- Tango2 (text-only T2A) achieves the highest mean concept accuracy on VinTAGe-Bench (55.57%) above VinTAGe, which the authors attribute to data quality and quantity.
- Existing joint text-video methods ReWaS and FoleyCrafter show high on-screen but very low off-screen accuracy, indicating strong visual bias.
Limitations
- Mixed offscreen sounds in VinTAGe-Bench are synthetic mixtures from VGGSound test clips, covering only 14 onscreen and 24 offscreen categories.
- Concept accuracy slightly trails text-only Tango2, attributed to limited data quality and quantity.
- Temporal metrics from prior work are unsuitable for mixed audio, so temporal alignment relies on subjective MOS-T.
Experimental setup
Datasets: VGGSound VinTAGe-Bench
Benchmarks: VinTAGe-Bench VGGSound
Metrics: FAD FID MKL AV alignment AT alignment concept accuracy MOS-Q MOS-F MOS-T ISc
Baselines: SpecVQGAN Seeing-and-Hearing Diff-Foley Make-an-Audio AudioLDM2 Tango2 ReWaS FoleyCrafter
Builds on: SiT Flow matching Tango2 Diff-Foley FoleyCrafter
Closest papers by meaning
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.90)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.90)
- Tell What You Hear From What You See - Video to Audio Generation Through Text (NeurIPS 2024 · similarity 0.90)
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.89)
- AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (ICML 2026 · similarity 0.88)
- Audio-Sync Video Generation with Multi-Stream Temporal Control (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.