Research map / Label-efficient and robust learning / Recognition and long-tail
Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes mini-batch Learning-to-match (m-LTM) with Mahalanobis ground metric and a partial-OT variant to learn audio-text joint embeddings that are more transferable and noise-tolerant than contrastive and triplet losses.
Problem
Learning joint audio-text embedding spaces for retrieval that close the modality gap and remain robust to misaligned (noisy) audio-caption pairs; conventional LTM does not scale to deep learning.
Approach
Adapts inverse optimal transport Learning-to-match to mini-batches with a Mahalanobis-enhanced ground metric (positive definite M updated by projected gradient) and uses partial OT to discard misaligned pairs from the transport plan.
Key results
- On AudioCaps (ResNet-38 encoder), m-LTM reaches R@1 39.10 text-to-audio and 49.94 audio-to-text vs contrastive 33.9 and 39.4.
- Zero-shot ESC-50 retrieval R@1 81.0 for m-LTM vs 72.25 for contrastive loss; modality gap 0.224 vs 0.922 on ESC50.
- At 60% noise on AudioCaps, m-LTM with POT gets R@1 27.73 (text-to-audio) vs 20.58 for contrastive loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use m-LTM instead of the strongest listed baseline | Audio-text retrieval · AudioCaps and Clotho | R@1: 36.7 -> 39.10; 45.3 -> 49.94; 15.79 -> 16.65; 17.42 -> 22.1 | Helped |
| Use m-LTM instead of contrastive loss | AudioCaps-pretrained model · ESC-50 test zero-shot sound event detection | R@1: 72.25 -> 81.0 | Helped |
| Use m-LTM instead of contrastive loss | Audio-text embedding · AudioCaps, Clotho, and ESC-50 | modality gap: 0.181 -> 0.117; 0.266 -> 0.142; 0.922 -> 0.224 | Helped |
| Add partial optimal transport to m-LTM | AudioCaps retrieval under noisy correspondence · 20%, 40%, and 60% noise | R@1: 35.51 -> 35.92; 46.64 -> 47.12; 32.58 -> 33.64; 40.31 -> 42.63; 25.26 -> 27.73; 34.08 -> 35.42 | Helped |
| Replace Euclidean distance with Mahalanobis distance (ablation) | m-LTM · AudioCaps retrieval | R@1: 38.13 -> 39.10; 48.15 -> 49.94 | Helped |
| Set epsilon to 0.1 instead of 0.05 (ablation) | m-LTM · AudioCaps retrieval | R@1: 39.10 -> 29.86; 49.94 -> 39.81 | Hurt |
| Set epsilon to 0.5 instead of 0.05 (ablation) | m-LTM · AudioCaps retrieval | R@1: 39.10 -> 9.59; 49.94 -> 10.55 | Hurt |
| Set epsilon to 0.01 (ablation) | m-LTM · AudioCaps text-to-audio retrieval | R@1: 39.10 -> 0.10 | Hurt |
| Set epsilon to 0.02 (ablation) | m-LTM · AudioCaps text-to-audio retrieval | R@1: 39.10 -> 0.17 | Hurt |
| Set epsilon to 0.03 instead of 0.05 (ablation) | m-LTM · AudioCaps text-to-audio retrieval | R@1: 39.10 -> 38.28 | Hurt |
| Set epsilon to 0.04 instead of 0.05 (ablation) | m-LTM · AudioCaps text-to-audio retrieval | R@1: 39.10 -> 37.22 | Hurt |
| Add entropic regularization to contrastive loss (ablation) | ResNet38 contrastive model · AudioCaps audio-to-text retrieval | R@1: 39.4 -> 44.11 | Helped |
Surprising findings
- Triplet loss collapses under 40% noise (R@1 0.1 text-to-audio), while contrastive loss remains relatively robust.
- Contrastive loss with entropic regularization gives only small gains over plain contrastive loss, far below m-LTM.
Limitations
- Baseline numbers are copied from reference papers rather than re-run under identical settings for some comparisons.
- Noise is synthetic (random shuffling) rather than real web-collected misalignment.
- Epsilon sensitivity: performance collapses for epsilon >= 0.1 and the transport mass must be tuned per noise level.
Experimental setup
Datasets: AudioCaps Clotho ESC-50
Benchmarks: AudioCaps Clotho ESC-50
Metrics: R@1 R@5 R@10 mAP modality gap norm
Baselines: contrastive loss triplet loss contrastive loss with entropic regularization
Builds on: learning-to-match sinkhorn partial optimal transport contrastive loss
Closest papers by meaning
- MATS: An Audio Language Model under Text-only Supervision (ICML 2025 · similarity 0.85)
- AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression (NeurIPS 2026 · similarity 0.85)
- Learning Audio-guided Video Representation with Gated Attention for Video-Text Retrieval (CVPR 2025 · similarity 0.85)
- Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.84)
- FLAM: Frame-Wise Language-Audio Modeling (ICML 2025 · similarity 0.84)
- Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.