Research map / Vision-language and multimodal / Cross-modal learning
An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-purposes the LLaVA-1.5 image captioner for zero-shot audio captioning by aligning audio (CAV-MAE) token distributions with image tokens via MMD or attentive optimal transport, plus prefix tuning and audiovisual distillation.
Problem
Audio captioning lacks large, high-quality audio-caption pairs, and contrastive audiovisual alignment leaves a modality gap that hinders swapping image tokens for audio tokens in a VLM.
Approach
Align the full token distribution of an audio encoder with the image encoder's tokens (via MMD or optimal transport with cross-attention weights) so the frozen image captioner can accept audio tokens, then distill audiovisual pseudo-captions into an audio-only MLP without any audio-caption supervision.
Key results
- On AudioCaps, DALI-Att-OT with audio+image input reaches SPIDEr 0.1946 vs 0.183 for supervised CLAP-based Salewski et al. and 0.1499 for image-only captioner.
- Audio-only DALI-Att-OT achieves SPIDEr 0.1592 on AudioCaps, well above Shaharabany et al. (ROUGE-L 0.082 reported for ImageBind).
- On Clotho, DALI-MMD reaches SPIDEr 0.0655 vs 0.097 for supervised CLAP-based Salewski et al., outperforming it on METEOR and ROUGE-L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add audiovisual distillation to DALI attentive OT (ablation) | DALI attentive OT · AudioCaps test | SPIDEr: 0.1443 -> 0.1592 | Helped |
| add audiovisual distillation to DALI MMD (ablation) | DALI MMD · AudioCaps test | SPIDEr: 0.1385 -> 0.1360 | Hurt |
| add audiovisual distillation to DALI OT (ablation) | DALI OT · AudioCaps test | SPIDEr: 0.1183 -> 0.1422 | Helped |
| add audiovisual distillation to contrastive alignment (ablation) | Contrastive alignment · AudioCaps test | SPIDEr: 0.0871 -> 0.1524 | Helped |
| add audiovisual distillation to DALI attentive OT (ablation) | DALI attentive OT · Clotho test | SPIDEr: 0.0355 -> 0.0625 | Helped |
| add audiovisual distillation to DALI MMD (ablation) | DALI MMD · Clotho test | SPIDEr: 0.0640 -> 0.0655 | Helped |
| add audiovisual distillation to DALI OT (ablation) | DALI OT · Clotho test | SPIDEr: 0.0299 -> 0.0574 | Helped |
| add audiovisual distillation to contrastive alignment (ablation) | Contrastive alignment · Clotho test | SPIDEr: 0.0377 -> 0.0620 | Helped |
| add image tokens alongside DALI attentive OT audio tokens | DALI attentive OT · AudioCaps test | SPIDEr: 0.1592 -> 0.1946 | Helped |
| increase prefix-tuning examples beyond 16 image-audio caption pairs | DALI audio captioning · Audio captioning | SPIDEr | No clear effect |
Surprising findings
- Contrastive alignment without distribution matching yields noisy captions, while distribution alignment keeps the audio tokens close to image tokens.
- Prefix tuning performance saturates at around 16 image-caption pairs, suggesting the prefix mainly specifies the task rather than teaching captioning.
Limitations
- Not robust to complete cross-modal mismatches where the image is unrelated to the audio.
- Requires some supervision to learn sounds that are not visible (e.g., wind), which images cannot describe.
- Audiovisual distillation on a filtered AudioSet subset; generalization to other data is not fully established, and no error bars reported.
Experimental setup
Datasets: AudioSet AudioCaps Clotho AudioSet balanced
Benchmarks: AudioCaps Clotho
Metrics: METEOR ROUGE-L SPIDEr BLEU4 CIDEr SPICE
Baselines: Shaharabany et al. (ImageBind + GPT-2) Salewski et al. (CLAP + OPT, ZerAuCaps) Contrastive alignment Image-only LLaVA captioner
Builds on: LLaVA-1.5 CAV-MAE CLIP Shaharabany et al. zero-shot audio captioning Maximum Mean Discrepancy Optimal Transport (EMD)
Closest papers by meaning
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.89)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.88)
- Aligned Better, Listen Better for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.87)
- OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning (NeurIPS 2026 · similarity 0.87)
- SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment (ICML 2024 · similarity 0.87)
- Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.