Research map / Detection and segmentation / Open-set and few-shot detection
Segment beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Segment Beyond View (SBV), an audio-visual teacher-student distillation model (Omni2Ego) that segments out-of-field-of-view vehicles from a limited-FoV view and binaural audio.
Problem
Partially missing modality in audio-visual semantic segmentation: a limited field-of-view camera misses objects outside its FoV (e.g., oncoming vehicles for AR/pedestrian safety), while only binaural audio is available at test time.
Approach
Distill panoramic visual information and 8-channel audio teachers into a student that only sees first-person view and binaural audio, using feature alignment, foreground-masked logits distillation, and modality reconstruction losses plus an audio-visual feature fusion attention module.
Key results
- SBV reaches 53.0 mIoU overall on AuditoryTestPseudo and 50.1 on AuditoryTestManual, versus 47.5 and 42.5 for TPAVI.
- On out-of-view regions SBV gets 46.7 mIoU (AuditoryTestPseudo) vs 39.0 for TPAVI, a 7.7-point gain.
- Monaural audio input lowers SBV's out-of-view mIoU by about 3.5 and 5.5 points on the two test sets while FPV performance barely changes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SBV instead of TPAVI | Audio-visual semantic segmentation · AuditoryTestPseudo OOV | mIoU (%): 39.0 -> 46.7 | Helped |
| Use SBV instead of TPAVI | Audio-visual semantic segmentation · AuditoryTestManual OOV | mIoU (%): 31.9 -> 43.3 | Helped |
| Use mono rather than binaural audio | SBV · AuditoryTestPseudo OOV | mIoU (%): 46.7 -> 43.2 | Hurt |
| Use mono rather than binaural audio | SBV · AuditoryTestManual OOV | mIoU (%): 43.3 -> 37.8 | Hurt |
| Add Omni2Ego distillation (ablation) | SBV · AuditoryTestPseudo OOV | mIoU (%): 43.5 -> 46.7 | Helped |
| Add Omni2Ego distillation (ablation) | SBV · AuditoryTestManual OOV | mIoU (%): 41.0 -> 43.3 | Helped |
| Add AVFFM (ablation) | SBV · AuditoryTestPseudo OOV | mIoU (%): 40.3 -> 46.7 | Helped |
| Add AVFFM (ablation) | SBV · AuditoryTestManual OOV | mIoU (%): 37.2 -> 43.3 | Helped |
| Add Modality Reconstruction Loss (ablation) | SBV · AuditoryTestPseudo All | mIoU (%): 45.9 -> 53.0 | Helped |
| Add Modality Reconstruction Loss (ablation) | SBV · AuditoryTestManual All | mIoU (%): 43.7 -> 50.1 | Helped |
Surprising findings
- Vision-only SegFormer on first-person view achieves only 12.7 mIoU out-of-view, showing visual input alone is insufficient for out-of-view objects.
Limitations
- Trained and tested only on urban traffic data from Zurich; may fail in non-urban scenes with different distributions.
- Not evaluated on videos from AR or pedestrian perspectives or in user studies.
- AuditoryTestManual contains only 80 manually labeled images.
Experimental setup
Datasets: Omni Auditory Perception Dataset Cityscapes
Benchmarks: AuditoryTestPseudo AuditoryTestManual
Metrics: mIoU F-score (beta=0.3)
Baselines: SegFormer SoundNet TPAVI SBV-V SBV-A
Builds on: SegFormer SoundNet TPAVI
Closest papers by meaning
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.87)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.87)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.86)
- Unsupervised Audio-Visual Segmentation with Modality Alignment (AAAI 2025 · similarity 0.86)
- Few-Shot Incremental Learning via Foreground Aggregation and Knowledge Transfer for Audio-Visual Semantic Segmentation (AAAI 2025 · similarity 0.86)
- AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.