Research map / Vision-language and multimodal / Cross-modal learning
Looking Similar Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses multilingual movie dubs as counterfactual audio pairs (same visuals, different speech) to augment SimCLR-style cross-modal audiovisual contrastive pretraining, improving HEAR and LVU downstream performance.
Problem
Cross-modal contrastive audiovisual representation learning treats scenes that look similar but have different speech as negatives, which may harm learned representations; counterfactual cross-modal pairs are rarely available.
Approach
Use dubbed movie audio tracks (same video, different spoken language, otherwise similar sound) as positive audio views in a cross-modal contrastive objective, combined with hierarchical same-title batch sampling for hard negatives, so the model learns to ignore speech content.
Key results
- Dub-augmented B.5 reaches 85.25 on HEAR ESC-50 and 69.16 on LVU Genre, with new SOTA on LVU Genre and Speaking Style among compared methods.
- Enforcing dub augmentation lowers VoxLingua107 and Speech Commands accuracy (e.g., A.2 vs A.1 drop), a modest linguistic trade-off.
- Multilingual pretraining without EN titles or with more language variation (B.3) gives gains on VoxLingua, with an absolute 5.6% gain reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dub augmentation to A.1 (ablation) | A.1 · HEAR sound and scene classification and non-semantic speech tasks | accuracy / mean average precision | Helped |
| Add dub augmentation to A.1 (ablation) | A.1 · LVU tasks | accuracy | Mixed |
| Add dub augmentation to A.3 (ablation) | A.3 · Speech Commands (Full) | accuracy: 49.53 -> 44.86 | Hurt |
| Add dub augmentation to A.3 (ablation) | A.3 · VoxLingua107 Top10 | accuracy: 57.65 -> 59.01 | Hurt |
| Add dub augmentation to B.3 (ablation) | B.3 · LVU tasks | accuracy | Helped |
| Add dub augmentation to B.3 (ablation) | B.3 · VoxLingua107 Top10 | accuracy: 60.20 -> 59.69 | Hurt |
| Use all-language pretraining rather than English-only pretraining (ablation) | B.1 · VoxLingua107 Top10 | accuracy: 57.14 -> 60.20 | Helped |
| Use all-language pretraining rather than English-only pretraining (ablation) | B.1 · Speech Commands (Full) | accuracy: 47.66 -> 50.47 | Helped |
| Add within-modal objectives (ablation) | B.3 with dub augmentation · HEAR tasks | accuracy / mean average precision | Mixed |
| Use MViT-B rather than MViT-S (ablation) | B.3, B.4, and B.5 · downstream tasks | accuracy / mean average precision | No clear effect |
Surprising findings
- Larger MViT-B (about 40% more parameters) did not give a meaningful boost over MViT-S in most tasks.
Limitations
- Dataset is proprietary and only possibly released for non-commercial use, limiting reproducibility.
- Dub augmentation slightly degrades language identification and keyword spotting performance.
- Synthetic counterfactual pipeline has limited synthetic quality.
Experimental setup
Datasets: in-house movie and TV pretraining dataset with multilingual dubs HEAR tasks LVU LVU-M (synthetic multilingual counterpart)
Benchmarks: HEAR LVU GTZAN
Metrics: top-1 accuracy mean average precision
Baselines: HEAR leaderboard best self-supervised GURA Fuse HuBERT CP-JKU PaSST CLAP Object Transformer (LVU) ViS4mer SVT SCALE STCA Movies2Scenes
Builds on: SimCLR MViT Movies2Scenes
Closest papers by meaning
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.88)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
- Asymmetric Hierarchical Anchoring for Robust Audio–Visual Cross-Modal Generalization (NeurIPS 2026 · similarity 0.87)
- Gramian Multimodal Representation Learning and Alignment (ICLR 2025 · similarity 0.86)
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.86)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.