Research map / Video / Video segmentation and tracking
Supervising Sound Localization by In-the-wild Egomotion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns binaural sound localization from in-the-wild camera egomotion (visual rotation and translation direction labels from SuperGlue/Perspective Fields) combined with IID cues, and introduces the StereoWalks dataset.
Problem
Stereo/binaural sound localization struggles in real-world videos because simulated training doesn't transfer and ground-truth direction labels are costly; the paper asks whether camera egomotion in unlabeled in-the-wild videos can supervise localization.
Approach
Use the sign of camera rotation and front-back translation estimated by off-the-shelf multi-view geometry (SuperGlue + Perspective Fields) as weak pseudo-labels that restrict the compatible angle pairs of two audio segments, combined with an IID cross-entropy loss, to train a ResNet-18 azimuth classifier on real internet videos.
Key results
- On YT-Stereo-iPhone, Ours-Full reaches 34.0 MAE and 61.7% 2clf versus 71.4 MAE and 54.0% for GTRot.
- On the in-the-wild binaural benchmark, Ours-full achieves 87.5% 2clf, close to the best StereoCRW (87.2%) and MonoCLR (87.4%) results.
- In simulated overlapping-source settings, Ours-Full outperforms GTRot in 2clf (87.2% vs 83.5%) on the Overlap setting but GTRot is best on one-source MAE (4.3 vs 9.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add egomotion rotation and translation losses to IID-only training (ablation) | Ours on YT-Stereo-iPhone · YT-Stereo-iPhone | MAE (°): 37.3 -> 34.0 | Helped |
| add egomotion rotation and translation losses to IID-only training (ablation) | Ours on YT-Stereo-iPhone · YT-Stereo-iPhone | 2clf (%): 54.7 -> 61.7 | Helped |
| add egomotion rotation and translation losses to IID-only training (ablation) | Ours on Stereo-Fountain · Stereo-Fountain | MAE (°): 88.3 -> 57.2 | Hurt |
| add egomotion rotation and translation losses to IID-only training (ablation) | Ours on In-the-wild audio · In-the-wild audio | 2clf (%): 87.7 -> 87.5 | No clear effect |
| add translation loss to rotation and IID losses (ablation) | Ours on YT-Stereo-iPhone · YT-Stereo-iPhone | 2clf (%): 55.4 -> 61.7 | Helped |
| add rotation loss to translation and IID losses (ablation) | Ours on YT-Stereo-iPhone · YT-Stereo-iPhone | 2clf (%): 53.4 -> 61.7 | Helped |
| add egomotion direction supervision instead of oracle rotation-angle supervision | Ours vs GTRot · HM3D-SS 2.0, Overlap | 2clf (%): 83.5 -> 87.2 | Helped |
| add IID loss to egomotion rotation and translation losses (ablation) | Ours on HM3D-SS 2.0 · HM3D-SS 2.0, Rotation&Translation | MAE (°): 14.7 -> 11.7 | Helped |
| remove translation loss from rotation and IID losses (ablation) | Ours on HM3D-SS 2.0 · HM3D-SS 2.0, Overlap R&T | 2clf (%): 81.4 -> 76.0 | Hurt |
| remove rotation loss from translation and IID losses (ablation) | Ours on HM3D-SS 2.0 · HM3D-SS 2.0, Overlap R&T | 2clf (%): 72.5 -> 76.0 | Helped |
Surprising findings
- Ours-Full trained on real videos does not generalize to simulated data, and the simulated-trained model does not generalize to real videos (sim2real gap).
- Removing the translation loss yields performance on par with the full model on YT-Stereo, suggesting relative translation can be disregarded for distant sources in walking tours.
Limitations
- Low-quality stereo audio and prevalent overlapping sounds complicate data cleaning.
- Filtering may overlook brief sound effects in unlabeled monocular videos.
- Front-back ambiguity for stereo devices; predictions are reflected to the front and only azimuth is estimated.
Experimental setup
Datasets: YT-Stereo YT-Stereo-iPhone Stereo-Fountain Binaural-Fountain HM3D-SS 2.0 In-the-wild audio (Chen et al. 2023) LibriSpeech Habitat-Matterport 3D
Benchmarks: StereoWalks (YT-Stereo-iPhone, Stereo-Fountain) In-the-wild audio binaural benchmark (Chen et al. 2023) HM3D-SS 2.0 simulated settings
Metrics: MAE (degrees) 2-way classification accuracy 8-way classification accuracy
Baselines: IID-direct GTRot Ours – IID only MonoCLR StereoCRW GCC-PHAT Supervised
Builds on: Chen et al. 2023 learning sound localization from camera rotation (GTRot) SuperGlue Perspective Fields ResNet-18
Closest papers by meaning
- Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos (CVPR 2024 · similarity 0.90)
- Cyclic Learning for Binaural Audio Generation and Localization (CVPR 2024 · similarity 0.87)
- Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues (CVPR 2025 · similarity 0.86)
- EgoSound: Benchmarking Sound Understanding in Egocentric Videos (CVPR 2026 · similarity 0.86)
- SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos (CVPR 2024 · similarity 0.86)
- Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.