Research map / Vision-language and multimodal / Cross-modal learning
From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VAB, a unified audio-visual encoder-generator that pre-trains a multiway transformer to predict masked audio codec tokens conditioned on frozen CLIP visual features, enabling fast parallel video-to-audio generation and downstream retrieval and classification.
Problem
Audio-visual representation learning and video-to-audio generation have been studied separately, and existing generative audio models are slow (autoregressive) or limited to specialized generation without unified representation capability.
Approach
Perform masked audio token prediction conditioned on visual features entirely in latent spaces (pre-trained neural codec tokens and frozen CLIP frame features) with a multiway transformer using modality-specific experts in lower layers and shared layers above, so the same pre-trained model supports iterative parallel decoding for generation and fine-tuning for retrieval and classification.
Key results
- VAB-Encodec reaches FAD 2.69 on VGGSound video-to-audio generation with 0.4 s per 10 s sample, about 17x faster than FoleyGen (6.9 s).
- VAB-Encodec achieves R@1 39.5 on AudioSet video-to-audio retrieval, roughly double the baselines (ImageBind 22.1).
- VAB-Encodec gets 91.4% on ESC-50, below MAViL (94.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train with masked audio token prediction before classification fine-tuning rather than train from scratch (ablation) | VAB-DAC-Test · VGGSound | accuracy, audio-visual: 54.7 -> 62.4 | Helped |
| add modal-specific experts to the first 12 layers (ablation) | VAB-DAC-Test · VGGSound | accuracy, audio-visual: 42.5 -> 62.4 | Helped |
| add contrastive fine-tuning after masked-token pre-training (ablation) | VAB-DAC-Test · VGGSound | accuracy, audio-visual: 56.2 -> 62.4 | Helped |
| use a 75% mean masking ratio instead of 55% (ablation) | VAB-DAC-Test · VGGSound classification and generation | accuracy / FAD: 56.2 -> 57.2; 5.30 -> 6.55 | Mixed |
| use a 35% mean masking ratio instead of 55% (ablation) | VAB-DAC-Test · VGGSound classification and generation | accuracy / FAD: 56.2 -> 55.2; 5.30 -> 6.04 | Mixed |
| replace eva-CLIP visual features with MAE features (ablation) | VAB-DAC-Test · VGGSound | accuracy, audio-visual: 62.4 -> 55.9 | Hurt |
| replace eva-CLIP visual features with MAE features (ablation) | VAB-DAC-Test · VGGSound | generation FAD: 4.18 -> 4.93 | Hurt |
| increase masking temperature from 4.5 to 25.5 (ablation) | VAB-Encodec · video-to-audio generation | FAD: 5.75 -> 2.69 | Helped |
| use 16 decoding iterations instead of 8 (ablation) | VAB-Encodec · video-to-audio generation | FAD: 3.52 -> 3.1 | Helped |
| increase decoding iterations from 16 to 36 (ablation) | VAB-Encodec · video-to-audio generation | FAD: 3.1 -> 3.25 | Hurt |
| use classifier-free guidance scale 5 rather than no guidance (ablation) | VAB-DAC · video-to-audio generation | FAD: 5.93 -> 3.24 | Helped |
| use masked-prediction then contrastive training rather than contrastive then masked prediction (ablation) | VAB-DAC-Test · VGGSound retrieval and generation | FAD / retrieval R@1: 4.14 -> 4.18; 27.7 -> 28.2; 32.5 -> 30.5 | Mixed |
Surprising findings
- Simultaneous contrastive and masked-prediction pre-training failed to converge, so the two objectives were applied sequentially.
Limitations
- Visual features are limited to 1 fps, so audio may not be perfectly synchronized with video.
- Frame-level pre-extracted CLIP embeddings omit spatial information.
- Audio-only classification lags best methods, and training used only single A100 GPUs with small batches.
Experimental setup
Datasets: AudioSet VGGSound ESC-50 Speech Commands
Benchmarks: VGGSound AudioSet MSR-VTT ESC-50 Speech Commands
Metrics: FAD KLD MOS R@1 R@5 R@10 accuracy mAP
Baselines: SpecVQGAN IM2WAV Diff-Foley FoleyGen ImageBind LanguageBind CAV-MAE MAViL
Builds on: MaskGIT VampNet Multiway Transformer (BEiT-3) CAV-MAE Encodec
Closest papers by meaning
- Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling (CVPR 2024 · similarity 0.89)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.89)
- Tell What You Hear From What You See - Video to Audio Generation Through Text (NeurIPS 2024 · similarity 0.88)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.88)
- Few-Shot Audio-Visual Class-Incremental Learning with Temporal Prompting and Regularization (AAAI 2025 · similarity 0.88)
- V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.