Research map / Vision-language and multimodal / Cross-modal learning
V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes V2A-Mapper, a lightweight diffusion- or regression-based mapper that translates CLIP image embeddings into CLAP audio space so frozen AudioLDM can generate sound from visual input.
Problem
Open-domain vision-to-audio generation, where existing systems are complex, separately trained pipelines built from scratch on modest datasets; the domain gap between CLIP visual and CLAP audio embeddings also blocks direct use of audio foundation models.
Approach
Freeze CLIP, CLAP and AudioLDM and train only a small mapper (diffusion-based Transformer, predicting the target CLAP embedding) that translates CLIP visual embeddings into the CLAP audio space, which AudioLDM then consumes as its conditioning.
Key results
- On VGGSound, the V2A-Mapper (diffusion Transformer) reaches FD 24.168 and CS 9.720 vs Im2Wav FD 51.500 and CS 7.827, with 48.83M trainable parameters vs 360.40M for Im2Wav.
- Compared with Im2Wav, the method is about 24x faster at inference (35.45 s vs 864.12 s for 100 samples) and reports 53% FD and 19% CS improvements.
- Regression MLP mapper gets a slightly higher CS (9.927) than the diffusion MLP (8.685), but a far worse FD (35.059 vs 28.803), supporting the generative mapper for diversity and fidelity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the V2A-Mapper with CLIP, CLAP, and AudioLDM instead of CLIPSonic-IQ | open-domain vision-to-audio generation · VGGSound | FD: 27.124 -> 24.168 | Helped |
| use the V2A-Mapper with CLIP, CLAP, and AudioLDM instead of Im2Wav | open-domain vision-to-audio generation · VGGSound | CS: 7.827 -> 9.720 | Helped |
| use the V2A-Mapper instead of CLIPSonic-IQ | open-domain vision-to-audio generation · ImageHear | CS: 11.392 -> 11.950 | Helped |
| add the V2A-Mapper rather than pass visual features directly to AudioLDM (ablation) | CLIP + AudioLDM · VGGSound | FD: 72.527 -> 24.168 | Helped |
| add the V2A-Mapper rather than pass visual features directly to AudioLDM (ablation) | CLIP + AudioLDM · VGGSound | CS: 5.258 -> 9.720 | Helped |
| add the V2A-Mapper rather than pass visual features directly to AudioLDM (ablation) | CLIP + AudioLDM · ImageHear | CS: 4.026 -> 11.950 | Helped |
| use BLIP captions as an intermediate bridge instead of the V2A-Mapper (ablation) | vision-to-audio generation with AudioLDM · VGGSound | FD: 24.168 -> 56.397 | Hurt |
| replace the generative diffusion mapper with regression MLP (ablation) | V2A-Mapper · VGGSound | FD: 24.168 -> 35.059 | Hurt |
| replace the generative diffusion mapper with regression MLP (ablation) | V2A-Mapper · VGGSound | CS: 9.720 -> 9.927 | Helped |
| use Transformer instead of MLP in the regression mapper (ablation) | regression V2A-Mapper · VGGSound | FD: 35.059 -> 29.378 | Helped |
| use Transformer instead of MLP in the generative mapper (ablation) | diffusion V2A-Mapper · VGGSound | CS: 8.685 -> 9.720 | Helped |
| average frame features instead of using a random frame (ablation) | V2A-Mapper video input · VGGSound | CS: 9.200 -> 9.720 | Helped |
Surprising findings
- The reported relevance (CS 9.720) on VGGSound exceeds the reference ground-truth CS (8.925), which the authors attribute to noisy VGGSound audio-visual correspondence.
Limitations
- No fine-grained controllability over sound details.
- Fails when visual cues are unclear (multiple objects, blurry or damaged images).
- Does not explicitly handle temporal alignment between audio and visual signals.
Experimental setup
Datasets: VGGSound ImageHear LAION-Audio-630K AudioSet AudioCaps Clotho Freesound BBC Sound Effects
Benchmarks: VGGSound ImageHear
Metrics: Fréchet Distance (FD) Fréchet Audio Distance (FAD) CLIP-Score (CS) MOS JMD inference time trainable parameters
Baselines: Im2Wav CLIPSonic-IQ vision-txt-audio (BLIP captioner + AudioLDM) Make-An-Audio
Builds on: AudioLDM CLIP CLAP DALL-E 2 prior Im2Wav
Closest papers by meaning
- PrismAudio: Decomposed Chain-of-Thought and Multi-dimensional Rewards for Video-to-Audio Generation (ICLR 2026 · similarity 0.88)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.88)
- X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation (NeurIPS 2026 · similarity 0.87)
- SonicVisionLM: Playing Sound with Vision Language Models (CVPR 2024 · similarity 0.87)
- AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs (ICML 2026 · similarity 0.87)
- ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.