Research map / Video / Video segmentation and tracking
Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Self-supervised masked audio-visual inpainting of binaural audio from egocentric video, with a channel-plus-token masking strategy, yielding spatial features that improve active speaker detection and spatial audio denoising.
Problem
Learning audio-visual representations that capture spatial correspondence between egocentric video and binaural audio for social settings, where existing methods focus on exocentric monaural audio and semantic cues.
Approach
Pretrain an audio-visual masked autoencoder to inpaint masked binaural audio tokens and full audio channels from video plus unmasked audio, using a mixed masking protocol (random token masking plus full-channel masking) so the encoder learns spatial cues; then fuse the frozen-or-finetuned features into ASD and denoising backbones.
Key results
- Active speaker detection: with TalkNet backbone, the method reaches 63.9 mAP on EgoCom and 71.8 on EasyCom, above the best alternate pretraining baseline (SSR++ at 70.6 on EasyCom and 2.5D-VS++ at 63.4 on EgoCom).
- Spatial audio denoising on EgoCom at 0 dB: SI-SDRi of 2.20 versus 2.11 for 2.5D-VS++ and 2.07 for AV-MAE; STFT distance 6.51 versus 6.60 for the next best.
- Training from scratch or with monaural-audio pretraining degrades both tasks, indicating the gains come from binaural spatial pretraining rather than semantic features alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed spatial audio-visual pretraining instead of 2.5D-VS++ features | TalkNet · EgoCom active speaker detection | mAP (%): 63.4 -> 63.9 | Helped |
| use the proposed spatial audio-visual pretraining instead of SSR++ features | TalkNet · EasyCom active speaker detection | mAP (%): 70.6 -> 71.8 | Helped |
| use the proposed spatial audio-visual pretraining instead of 2.5D-VS++ features | SPELL · EgoCom active speaker detection | mAP (%): 65.1 -> 65.6 | Helped |
| use the proposed spatial audio-visual pretraining instead of SSR++ features | SPELL · EasyCom active speaker detection | mAP (%): 67.4 -> 70.2 | Helped |
| use the proposed spatial audio-visual pretraining instead of 2.5D-VS++ features | U-Net · EgoCom, 0 dB noise | SI-SDRi: 2.11 -> 2.20 | Helped |
| use the proposed spatial audio-visual pretraining instead of 2.5D-VS++ features | U-Net · EgoCom, 0 dB noise | STFT distance: 6.60 -> 6.51 | Helped |
| add self-supervised pretraining (ablation) | TalkNet · EgoCom active speaker detection | mAP (%): 62.7 -> 63.9 | Helped |
| add self-supervised pretraining (ablation) | TalkNet · EasyCom active speaker detection | mAP (%): 62.9 -> 71.8 | Helped |
| add binaural rather than monaural audio during pretraining (ablation) | TalkNet · EgoCom active speaker detection | mAP (%): 61.0 -> 63.9 | Helped |
| add binaural rather than monaural audio during pretraining (ablation) | TalkNet · EasyCom active speaker detection | mAP (%): 69.4 -> 71.8 | Helped |
| add binaural rather than monaural audio during pretraining (ablation) | SPELL · EgoCom active speaker detection | mAP (%): 63.9 -> 65.6 | Helped |
| add binaural rather than monaural audio during pretraining (ablation) | SPELL · EasyCom active speaker detection | mAP (%): 69.0 -> 70.2 | Helped |
Surprising findings
- Pretraining with monaural audio instead of binaural lowers performance on both ASD and denoising, so the benefit is tied to spatial cues rather than general audio-visual semantics.
- Only EgoCom is used for denoising because mixing clips in EasyCom yields spatially overlapping sources at the same physical location.
Limitations
- ASD performance declines under drastic camera-wearer motion or heavy speech overlap.
- Denoising struggles when noise is semantically or acoustically similar to the target or when speakers are occluded or out of view.
- Denoising evaluated only on EgoCom with synthetic noise mixtures; EasyCom excluded as ill-posed.
Experimental setup
Datasets: EgoCom EasyCom
Benchmarks: EgoCom EasyCom
Metrics: mean average precision STFT distance SI-SDRi
Baselines: TalkNet SPELL Telling Left from Right (TLR) 2.5D Visual Sounds (2.5D-VS) Stereo sound ranking (SSR) AV-MAE U-Net ImageNet pretraining
Builds on: masked auto-encoding AV-MAE TalkNet SPELL U-Net
Closest papers by meaning
- Supervising Sound Localization by In-the-wild Egomotion (CVPR 2025 · similarity 0.90)
- SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos (CVPR 2024 · similarity 0.89)
- EgoSound: Benchmarking Sound Understanding in Egocentric Videos (CVPR 2026 · similarity 0.89)
- Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues (CVPR 2025 · similarity 0.88)
- The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective (CVPR 2024 · similarity 0.87)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.