Research map / Video / Video segmentation and tracking
Spherical World-Locking for Audio-Visual Localization in Egocentric Videos
Heeseung Yun, Ruohan Gao, Ishwarya Ananthabhotla, Anurag Kumar, Jacob Donley, Chao Li, Gunhee Kim, Vamsi Krishna Ithapu, Calvin Murdock
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MuST with spherical world-locking instead of AVLN/MAVASL | egocentric active speaker localization · EasyCom | mAP: 86.32 -> 89.88 | Improved |
| add head pose information to MuST (ablation) | MuST · EasyCom | mAP: 87.76 -> 89.88 | Improved |
| add rotation quaternion spatial similarity (ablation) | MuST · EasyCom | mAP: 88.83 -> 89.88 | Improved |
| add pose information to audio-only spherical localization (ablation) | MuST · RLR-CHAT | MAE g->p (degrees): 29.33 -> 15.23 | Improved |
| use multichannel instead of mono audio (ablation) | MuST with pose and visual · EasyCom | mAP: 73.47 -> 89.88 | Improved |
| add visual input on top of multichannel audio and pose (ablation) | MuST · EasyCom | mAP: 89.50 -> 89.88 | Improved |
| add modality-wise layer norm and attention projections (ablation) | MuST · EasyCom | mAP: 88.53 -> 89.88 | Improved |
| add modality-wise MLP on top of M-LN and M-Attn variants (ablation) | MuST · EasyCom | mAP: 89.67 -> 89.16 | Hurt |
| lengthen temporal window from 100ms to 300ms (ablation) | MuST · EasyCom | mAP: 82.96 -> 89.88 | Improved |
| use sparse grid decoding instead of sparse point decoding (ablation) | MuST · EasyCom | mAP: 88.95 -> 89.88 | Improved |
| jointly train all behavior anticipation tasks versus single-task (ablation) | MuST AVB · AEA gaze 700ms | MAE: 15.02 -> 14.75 | Improved |
| use audio-visual input without previous behavior context for trajectory (ablation) | MuST AV · AEA trajectory 300ms | MAE: 9.77 -> 92.38 | Hurt |
Closest papers by meaning
- Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos (CVPR 2024 · similarity 0.88)
- Supervising Sound Localization by In-the-wild Egomotion (CVPR 2025 · similarity 0.87)
- EgoSound: Benchmarking Sound Understanding in Egocentric Videos (CVPR 2026 · similarity 0.87)
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.86)
- Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues (CVPR 2025 · similarity 0.85)
- SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
