Research map / Video / Video generation
Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mel-QCD, which decomposes mel-spectrograms into quantized semantic codes plus continuous energy and std signals, predicts them from video with a V2X predictor, and feeds them via ControlNet and textual inversion into Auffusion for video-to-audio generation.
Problem
Video-to-audio generation via control signals extracted from video for text-to-audio diffusion models faces a trade-off: signals easy to predict from video lose control precision, while complete signals are hard to predict.
Approach
Decompose the mel-spectrogram per time slot into energy, standard deviation, and normalized shape; quantize the normalized semantic shape into a finite codebook (classification) while keeping energy and std as continuous regression targets, then recompose into a control map for ControlNet on a T2A diffusion model, plus textual inversion from video CLIP embeddings.
Key results
- Mel-QCD achieves FID 11.73, MKL 2.96, and Class ACC 45.91 on VGGSound, the best among compared methods on most metrics, versus FoleyCrafter's 13.11 FID and 31.54 Class ACC.
- Mel-QCD with textual inversion reaches FID 57.36, MKL 1.53, Class ACC 69.33 in the AvSync15 ablation, versus 65.10/2.64/52.00 for the base model without MQ or TI.
- Onset ACC for Mel-QCD (25.42) is second-best, behind VTA-LDM (26.83), and IB-AV (0.31) trails Seeing-and-Hearing (0.38).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Mel-QCD instead of the strongest listed baseline | Video-to-audio generation on VGGSound · VGGSound test | Class ACC: 31.54 -> 45.91 | Helped |
| Use Mel-QCD instead of the strongest listed baseline | Video-to-audio generation on VGGSound · VGGSound test | Onset ACC: 26.83 -> 25.42 | Hurt |
| Use Mel-QCD instead of the strongest listed baseline | Video-to-audio generation on VGGSound · VGGSound test | IB-AV: 0.38 -> 0.31 | Hurt |
| Add decomposition to the base model (ablation) | Mel-QCD · AvSync15 | FID: 65.10 -> 61.97 | Helped |
| Add discrete quantization to decomposition (ablation) | Mel-QCD · AvSync15 | FID: 61.97 -> 61.44 | Helped |
| Add SVQ to decomposition and discrete quantization (ablation) | Mel-QCD · AvSync15 | Class ACC: 58.67 -> 64.67 | Helped |
| Add textual inversion to Mel-QCD (ablation) | Mel-QCD · AvSync15 | Class ACC: 64.67 -> 69.33 | Helped |
| Use textual inversion without Mel-QCD components (ablation) | Base model · AvSync15 | FID: 65.10 -> 75.65 | Hurt |
| Increase SVQ codebook complexity across tested settings (ablation) | Mel-QCD · AvSync15 | FID: 53.09; 86.81 | Mixed |
| Use temporal smoothing or temporal mean instead of repetition (ablation) | Mel-QCD signal downsampling · AvSync15 | FID: 47.87 -> 52.43; 47.87 -> 53.48 | Hurt |
| Use alternative frequency compression strategies instead of DS-Repeat (ablation) | Mel-QCD signal compression · AvSync15 | FID: 47.57 -> 50.04; 47.57 -> 51.50; 47.57 -> 58.01 | Hurt |
| Increase pseudo-word tokens from 1 to 32 (ablation) | Textual inversion · AvSync15 | FID: 110.21 -> 75.65 | Helped |
Surprising findings
- Predicted Mel-QCD matches low-resolution mel in completeness while being far less complex, and outperforms onset and energy signals as control.
- Simply repeating (nearest-interpolating) the downsampled frequency bins works better than sparse zero-padding recovery.
Limitations
- VTA-LDM and Seeing-and-Hearing outperform on Onset ACC and IB-AV respectively, so the method is not best on all metrics
- Evaluation limited to VGGSound and a 1.1k test split; ReWaS excluded because its code could not reproduce results
- Predicted Mel-QCD has local timing and semantic deviations that textual inversion only partially corrects
Experimental setup
Datasets: VGGSound AvSync15
Benchmarks: VGGSound
Metrics: FID MKL Class ACC W-Dis JS-Div Onset ACC IB-AA IB-AV
Baselines: Auffusion SpecVQGAN Im2Wav DiffFoley VTA-LDM Seeing-and-Hearing FoleyCrafter
Builds on: Auffusion ControlNet FoleyCrafter Textual Inversion EnCodec
Closest papers by meaning
- Audio-Sync Video Generation with Multi-Stream Temporal Control (NeurIPS 2025 · similarity 0.87)
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.86)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.86)
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.85)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.85)
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.