Research map / Video / Video-language understanding
Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses audio as a bridge for egocentric-exocentric video alignment, adding a sound-vision cross-attention module and an LLM-generated sound-caption text module that jointly anchor both views to a sound-aware text space.
Problem
Aligning egocentric and exocentric videos fails when visual content differs substantially between views and some actions (e.g., laughing, sneezing) have no visible cues in the egocentric view.
Approach
Treat audio as a view-consistent bridge: contextualize audio with visual features via cross-attention and with text via LLM-generated closed-caption sound descriptions, then contrastively align both egocentric and exocentric sound-aware visual features to a sound-aware text anchor.
Key results
- On EgoExoLearn, SoundBridge reaches 50.50% average Ego-Exo retrieval top-1 vs 47.15% for EgoExo, the first method above 50% on that dataset.
- On CharadesEgo, SoundBridge reaches 74.65% average top-1 retrieval vs 65.19% for EgoExo.
- Egocentric scene recognition on CharadesEgo reaches 51.06%, and exocentric action recognition on EgoExoLearn reaches 56.25%, both above prior baselines (e.g., EgoInstructor 46.51% and 47.50%), though below VideoMAEv2 (62.50%) on action recognition.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sound-text module to EgoExo (ablation) | EgoExo · EgoExoLearn Ex2Eg | top-1 accuracy: 45.30 -> 50.18 | Helped |
| add sound-text module to EgoExo (ablation) | EgoExo · CharadesEgo Eg2Ex | top-1 accuracy: 68.32 -> 72.81 | Helped |
| add sound-vision module to EgoExo (ablation) | EgoExo · EgoExoLearn Eg2Ex | top-1 accuracy: 49.00 -> 48.46 | Hurt |
| add sound-vision module to EgoExo (ablation) | EgoExo · CharadesEgo Eg2Ex | top-1 accuracy: 68.32 -> 74.35 | Helped |
| add both sound-text and sound-vision modules to EgoExo (ablation) | EgoExo · EgoExoLearn Eg2Ex | top-1 accuracy: 49.00 -> 50.27 | Helped |
| add both sound-text and sound-vision modules to EgoExo (ablation) | EgoExo · CharadesEgo Ex2Eg | top-1 accuracy: 62.06 -> 71.63 | Helped |
| use GPT-3.5 rather than LLaMA3 to generate audio descriptions (ablation) | SoundBridge · EgoExoLearn Eg2Ex | top-1 accuracy: 50.27 -> 49.27 | Hurt |
| use GPT-3.5 rather than LLaMA3 to generate audio descriptions (ablation) | SoundBridge · CharadesEgo Eg2Ex | top-1 accuracy: 77.66 -> 73.76 | Hurt |
| use original caption as anchor instead of the sound-text module (ablation) | SoundBridge · EgoExoLearn Eg2Ex | top-1 accuracy: 50.27 -> 48.64 | Hurt |
| use sound caption as anchor instead of the sound-text module (ablation) | SoundBridge · EgoExoLearn Eg2Ex | top-1 accuracy: 50.27 -> 47.09 | Hurt |
| use sound feature as anchor instead of the sound-text module (ablation) | SoundBridge · EgoExoLearn Eg2Ex | top-1 accuracy: 50.27 -> 36.67 | Hurt |
Surprising findings
- Using sound captions or audio features alone as the anchor performs worse than the original text anchor, indicating audio needs textual context.
- Action recognition is below VideoMAEv2 (62.50%) despite the sound-aware representation, so the method does not beat the strongest task-specific model on that task.
Limitations
- Audio alone is not discriminative; the method depends on LLM-generated sound captions whose quality varies (GPT-3.5 gave lower results than Llama3).
- Action recognition on EgoExoLearn remains below VideoMAEv2 despite the sound-aware features.
Experimental setup
Datasets: CharadesEgo EgoExoLearn
Benchmarks: EgoExoLearn CharadesEgo
Metrics: top-1 accuracy
Baselines: FrozenInTime LaViLa EgoVLP InternVideo EgoExo EgoInstructor VideoMAEv2 InternImage
Builds on: EgoExo EgoInstructor CLIP Eclipse
Closest papers by meaning
- SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos (CVPR 2024 · similarity 0.90)
- EgoSound: Benchmarking Sound Understanding in Egocentric Videos (CVPR 2026 · similarity 0.89)
- Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos (CVPR 2024 · similarity 0.88)
- The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective (CVPR 2024 · similarity 0.88)
- Exocentric-to-Egocentric Video Generation (NeurIPS 2024 · similarity 0.87)
- Supervising Sound Localization by In-the-wild Egomotion (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.