Research map / Vision-language and multimodal / Cross-modal learning
SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns how human actions sound from narrated egocentric video via a multimodal contrastive-consensus loss (MC3) over audio, vision, and language, evaluated on sounding-action discovery/retrieval (Ego4D) and audio classification (EPIC-Sounds).
Problem
Learning which camera-wearer actions produce sound from in-the-wild narrated egocentric video without audio labels, where audio-visual correspondence alone captures general correlations rather than action-specific sounds.
Approach
Two-stage align-then-refine training: pairwise InfoNCE contrastive losses across audio, vision, and language, plus an intra-sample consensus loss that pushes each non-anchor modality's similarity to the anchor toward the minimum (bottleneck) pairwise similarity, so embeddings align only when all modalities agree.
Key results
- Sounding action discovery: MC3 reaches 0.598 ROC AUC for audio-visual vs 0.554 for ImageBind, and 0.658 for audio-language vs 0.642 for ImageBind.
- Sounding action retrieval: MC3 achieves Recall@10 of 88.5 for audio-to-video... (V→A R@10 72.8 vs 61.5 for ImageBind).
- EPIC-Sounds fine-tuning: MC3 reaches 55.97 Top-1 vs 53.47 for SSAST; linear probe 42.44 Top-1 vs 28.74 for SSAST.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multimodal consensus loss (stage two) to contrastive MC3 training (ablation) | MC3 embedding · Ego4D sounding action discovery | ROC AUC (AV): 0.598 vs 0.563 without consensus loss | Helped |
| Remove the align (contrastive pre-training) stage and optimize MC3 loss directly (ablation) | MC3 embedding · Ego4D sounding action discovery | ROC AUC (AV): 0.598 to 0.448 | Hurt |
| Remove contrastive loss in stage two (keep only consensus) (ablation) | MC3 embedding · Ego4D sounding action discovery | ROC AUC (AV): 0.598 to 0.436 | Hurt |
| Multimodal consensus coding plus contrastive coding vs. contrastive-only baselines (CMC, ImageBind) | audio-visual-language embedding · Ego4D sounding action discovery | ROC AUC (AV): 0.598 vs 0.550 (CMC) and 0.554 (ImageBind) | Helped |
| Multimodal consensus coding vs. CLAP-style and joint baselines for audio-language discovery | audio-language embedding · Ego4D sounding action discovery | ROC AUC (AL): 0.658 vs 0.637 (CLAP) and 0.642 (ImageBind) | Helped |
| Multimodal consensus coding vs. baselines for video-to-audio and language-to-audio retrieval | MC3 embedding · Ego4D sounding action retrieval | Recall@5 (L->A): 46.2 vs 49.8 for CLAP? no: 46.2 vs 44.1 (CMC), 42.6 (ImageBind); CLAP 49.8 is higher | Helped |
| Multimodal consensus coding vs. contrastive baselines for audio-visual retrieval | MC3 embedding · Ego4D sounding action retrieval | Recall@5 (V->A): 38.4 vs 36.5 (CMC), 32.8 (ImageBind) | Helped |
| Multimodal consensus coding vs. SSAST and ASF baselines, linear probe | MC3 audio encoder · EPIC-Sounds classification | Top-1 (linear probe): 42.44 vs 28.74 (SSAST) but below 45.53 (ASF, supervised pretraining) | Mixed |
| Multimodal consensus coding vs. SSAST and ASF baselines, fine-tuning | MC3 audio encoder · EPIC-Sounds classification | Top-1 (fine-tuning): 55.97 vs 53.75 (ASF) and 53.47 (SSAST) | Helped |
Surprising findings
- Only about half (17,693 of 33,000) of the in-the-wild narrated clips are sounding actions, underscoring the difficulty of distinguishing sounding from non-sounding visible actions.
- Directly optimizing the full MC3 loss without the align stage fails badly, showing the two-stage schedule is necessary for stable training.
Limitations
- Assumes temporally aligned narrated clips; future work needed for asynchronous multimodal streams.
- Evaluation of sounding actions relies on a newly collected annotation set of 33K clips only for Ego4D.
- Relies on a choice of anchor modality (audio) and scaling hyperparameters tuned on validation.
Experimental setup
Datasets: Ego4D EPIC-Sounds
Benchmarks: Ego4D sounding action annotations EPIC-Sounds
Metrics: ROC AUC PR AUC Recall@5 Recall@10 Top-1 Top-5 mCA MAP mAUC
Baselines: CLAP CM-ACC CMC ImageBind SSAST ASF
Builds on: InfoNCE CMC ImageBind CLAP
Closest papers by meaning
- Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues (CVPR 2025 · similarity 0.90)
- Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos (CVPR 2024 · similarity 0.89)
- EgoSound: Benchmarking Sound Understanding in Egocentric Videos (CVPR 2026 · similarity 0.88)
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.87)
- X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization (CVPR 2024 · similarity 0.87)
- Supervising Sound Localization by In-the-wild Egomotion (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.