Research map / Video / Video segmentation and tracking
Audio-Visual Camera Pose Estimation with Passive Scene Sounds and In-the-Wild Video
Ikechukwu D Adebi, Sagnik Majumder, Kristen Grauman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add spatial audio (DOA + binaural) to vision-only Reloc3r | Reloc3r · Ego-Exo4D val | Total AUC@5: 0.57 -> 0.81 | Improved |
| add spatial audio (DOA + binaural) to vision-only Reloc3r | Reloc3r · Ego-Exo4D val | Total AUC@20: 7.33 -> 8.82 | Improved |
| add DOA spectrum alongside binaural embedding (ablation) | Ours · Ego-Exo4D val | Total AUC@20: 8.41 -> 8.82 | Improved |
| add binaural embedding alongside DOA spectrum (ablation) | Ours · Ego-Exo4D val | Total AUC@20: 8.34 -> 8.82 | Improved |
| use learned binauralized embedding instead of monaural embedding (ablation) | Ours · Ego-Exo4D val | Total AUC@5: 0.74 -> 0.81 | Improved |
| use the proposed fusion instead of audio-visual cross-attention | Reloc3r-AV Cross-Att · Ego-Exo4D val | Total AUC@20: 8.34 -> 8.82 | Improved |
| add spatial audio under corrupted vision | Reloc3r · Ego-Exo4D val (corrupted vision) | Total AUC@5: 0.21 -> 0.35 | Improved |
| add spatial audio under corrupted vision | Reloc3r · HM3D-SS test (corrupted vision) | rotation MAE (deg): 3.83 -> 2.46 | Improved |
| add spatial audio on clean simulated video | Reloc3r · HM3D-SS test | rotation MAE (deg): 0.09 -> 0.06 | Improved |
| add IMU input to the audio-visual model (ablation) | Ours · Ego-Exo4D val | Total AUC@5: 0.81 -> 0.85 | Improved |
| replace synchronized audio with cross-scene audio (ablation) | Ours · Ego-Exo4D val | Total AUC@5: 0.81 -> 0.50 | Hurt |
| replace synchronized audio with same-scene shuffled audio (ablation) | Ours · Ego-Exo4D val | Total AUC@5: 0.81 -> 0.66 | Hurt |
Closest papers by meaning
- Supervising Sound Localization by In-the-wild Egomotion (CVPR 2025 · similarity 0.86)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.86)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.85)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.85)
- Towards Audio-Visual Navigation in Noisy Environments: A Large-Scale Benchmark Dataset and an Architecture Considering Multiple Sound-Sources (AAAI 2025 · similarity 0.85)
- SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
