Research map / Vision-language and multimodal / Cross-modal learning
Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an unbiased sliced Wasserstein RBF kernel with rotary positional embedding, used as a training loss and reranking score with stochastic decoding (ACUS framework) to mitigate exposure bias and caption degeneration in audio captioning.
Problem
Encoder-decoder audio captioning models trained with teacher forcing suffer exposure bias and caption degeneration at inference; existing contrastive cosine-similarity reranking ignores temporal structure between audio and caption representations.
Approach
Define an unbiased sliced Wasserstein RBF kernel (expectation of exp of projected Wasserstein distance over projection directions) on position-augmented (rotary PE) audio and caption hidden sequences; use it as a temporal-similarity term in training and as a reranking score over stochastically decoded candidates.
Key results
- On AudioCaps, Enclap + ACUS reaches SPIDEr 0.500 and CIDEr 0.807 versus 0.480 and 0.770 for Enclap with beam search.
- On MMAU test mini, GAMA w/ USW-RBF raises average accuracy from 30.1% to 34.10%.
- Enclap + ACUS improves AudioCaps text-to-audio retrieval R@1 from 29.2 to 32.2 and caption length from 7.52 to 8.66.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add USW-RBF training and stochastic reranking | Enclap · AudioCaps test | CIDEr: 0.768 -> 0.807 | Helped |
| Add USW-RBF training and stochastic reranking | Enclap · Clotho test | CIDEr: 0.405 -> 0.419 | No clear effect |
| Add USW-RBF reranking | Enclap · AudioCaps text-to-audio retrieval | R@1: 30.4 -> 32.2 | Helped |
| Fine-tune GAMA with USW-RBF instead of contrastive learning | GAMA · MMAU test mini | Average accuracy: 30.49 -> 34.10 | Helped |
| Replace USW-RBF similarity with DTW (ablation) | ACUS with Enclap · AudioCaps test | CIDEr: 0.807 -> 0.762 | Hurt |
| Replace USW-RBF similarity with soft-DTW (ablation) | ACUS with Enclap · AudioCaps test | CIDEr: 0.807 -> 0.764 | Hurt |
| Replace USW-RBF with Wasserstein distance and positional embedding (ablation) | ACUS with Enclap · AudioCaps test | CIDEr: 0.807 -> 0.756 | Hurt |
| Remove positional embedding (ablation) | ACUS with Enclap · AudioCaps test | METEOR: 0.259 -> 0.262 | Hurt |
| Replace rotary PE with absolute PE (ablation) | ACUS with Enclap · AudioCaps test | METEOR: 0.26 -> 0.262 | Hurt |
| Use USW-RBF only during training, not reranking (ablation) | Enclap · AudioCaps test | SPIDEr: 0.48 -> 0.492 | Helped |
| Use USW-RBF in both training and inference rather than training only (ablation) | Enclap · AudioCaps test | SPIDEr: 0.492 -> 0.5 | Helped |
| Reduce projection count from L=50 to L=1 (ablation) | ACUS with Enclap · AudioCaps test | SPIDEr: 0.5 -> 0.491 | Hurt |
Surprising findings
- Only using the USW-RBF kernel in training gives small gains; the large improvement requires applying it at inference reranking too.
- Temperature sampling gives a slight drop in CIDEr, and the paper notes sampling hyperparameters matter.
Limitations
- Inference is slower because about 30 candidate captions must be sampled and reranked.
- Gains on Clotho for automated metrics like ROUGE-L and CIDEr are insignificant.
Experimental setup
Datasets: AudioCaps Clotho CompA-R MMAU
Benchmarks: AudioCaps Clotho CompA-R-test MMAU test mini
Metrics: METEOR ROUGE-L CIDEr SPICE SPIDEr caption length lexical diversity text-to-audio retrieval R@1/R@5/R@10 GPT4-o score accuracy
Baselines: LHDFF CNN14-GPT2 BART-tags Pengi AL-MixGen WavCaps AutoCap Enclap + CL CLIP-AAC MAAC GAMA w/ CL
Builds on: SW-RBF kernel contrastive search contrastive learning for text generation (CL regularization) SimCLR infoNCE
Closest papers by meaning
- An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching (NeurIPS 2024 · similarity 0.87)
- MATS: An Audio Language Model under Text-only Supervision (ICML 2025 · similarity 0.86)
- BRACE: A Benchmark for Robust Audio Caption Quality Evaluation (NeurIPS 2025 · similarity 0.86)
- OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning (NeurIPS 2026 · similarity 0.85)
- MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks (ICML 2026 · similarity 0.85)
- Audio Entailment: Assessing Deductive Reasoning for Audio Understanding (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.