Research map / Video / Video generation
Read, Watch and Scream! Sound Generation from Text and Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReWaS, which predicts a continuous energy control signal from video and feeds it with a text prompt into a ControlNet-style energy adapter on AudioLDM to generate temporally aligned audio.
Problem
Video-to-audio generation has limited quality and temporal alignment and cannot take text control, while text-to-audio models lack temporal understanding of video and timestamp-based controls are costly to annotate.
Approach
Use video to predict a continuous, smoothed frame-level energy (mean mel-spectrogram magnitude) as a structural control, and inject it into a pretrained AudioLDM via a ControlNet-style energy adapter, with the text prompt supplying semantic content.
Key results
- On VGGSound, ReWaS achieves best FD (15.24), FAD (2.16), energy MAE (0.1149), and AV-align (0.3008) among compared methods, with 204M trainable parameters versus 859M for Diff-Foley.
- On Greatest Hits, ReWaS reaches the best AP (63.28) and MAE (0.1398), though CondFoleyGen has higher onset accuracy (23.94 vs 19.15).
- In a 5-point MOS user study, ReWaS scores 3.70 audio quality, 4.04 relevance, and 3.68 temporal alignment, above the V2A baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ReWaS instead of V2A baselines | VGGSound · VGGSound | FD: 16.87 -> 15.24 | Helped |
| use ReWaS instead of V2A baselines | VGGSound · VGGSound | FAD: 5.57 -> 2.16 | Helped |
| use ReWaS instead of V2A baselines | VGGSound · VGGSound | AV-align: 0.2059 -> 0.3008 | Helped |
| use ReWaS instead of Seeing&Hearing | VGGSound · VGGSound | CLAP: 0.5805 -> 0.4353 | Hurt |
| use ReWaS instead of CondFoleyGen | Greatest Hits · Greatest Hits | AP: 60.24 -> 63.28 | Helped |
| use ReWaS instead of CondFoleyGen | Greatest Hits · Greatest Hits | Acc: 23.94 -> 19.15 | Hurt |
| use ReWaS instead of V2A baselines | VGGSound human evaluation · VGGSound human evaluation | audio quality MOS: 2.97 -> 3.70 | Helped |
| replace estimated video energy with ground-truth audio energy (ablation) | ReWaS · VGGSound | FD: 15.24 -> 13.93 | Helped |
| replace AudioLDM-M with Make-An-Audio backbone (ablation) | ReWaS · VGGSound | FAD: 2.16 -> 10.91 | Hurt |
| remove text prompt (ablation) | ReWaS · VGGSound ablation | CLAP: 0.44 -> 0.236 | Hurt |
| shuffle text prompts across video categories (ablation) | ReWaS · VGGSound ablation | CLAP: 0.44 -> 0.237 | Hurt |
| remove visual energy control (ablation) | ReWaS · VGGSound ablation | AV-Align: 0.30 -> 0.10 | Hurt |
Surprising findings
- Seeing&Hearing gets higher MKL and CLAP scores than ReWaS, indicating strong text dependence rather than video alignment.
- Removing the energy control drops AV-align from 0.30 to 0.10 in the ablation, while removing the text prompt drops CLAP sharply.
Limitations
- Inherits AudioLDM hallucinations; 18% of 100 sampled outputs flagged with hallucination in a small human check.
- Onset accuracy on Greatest Hits is lower than CondFoleyGen.
- Evaluation relies on 5-second clips and a partly small-scale MOS study with heterogeneous annotator channels.
Experimental setup
Datasets: VGGSound Greatest Hits
Benchmarks: VGGSound Greatest Hits
Metrics: FD FAD MKL CLAP score energy MAE AV-align onset accuracy average precision MOS audio quality MOS relevance
Baselines: SpecVQGAN Im2wav Diff-Foley Seeing&Hearing CondFoleyGen
Builds on: AudioLDM ControlNet SynchFormer Make-An-Audio
Closest papers by meaning
- Tell What You Hear From What You See - Video to Audio Generation Through Text (NeurIPS 2024 · similarity 0.90)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.90)
- SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation (ICLR 2025 · similarity 0.89)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.89)
- Audio Generation with Multiple Conditional Diffusion Model (AAAI 2024 · similarity 0.89)
- Animate and Sound an Image (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.