Research map / Vision-language and multimodal / Cross-modal learning
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Andrew Owens, Alexei A. Efros
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add self-supervised audio-visual pretraining (ablation) | Multisensory network · UCF-101 split 1 action recognition | accuracy (%): 68.1% -> 82.1% | Improved |
| add audio input to the pretrained network (ablation) | Multisensory network · UCF-101 split 1 action recognition | accuracy (%): 77.6% -> 82.1% | Improved |
| use raw waveform instead of spectrogram audio (ablation) | Multisensory network · UCF-101 split 1 action recognition | accuracy (%): 81.1% -> 82.1% | No clear effect |
| use misalignment pretext task instead of random audio-video pairing (ablation) | Multisensory network · UCF-101 split 1 action recognition | accuracy (%): 78.7% -> 82.1% | Improved |
| use multisensory features instead of I3D Kinetics features (ablation) | On/off separation model · VoxCeleb separation (all) | On/off L1 error: 12.3 -> 11.4 | Improved |
| use multiple frames instead of a single frame (ablation) | On/off separation model · VoxCeleb separation (all) | On/off L1 error: 14.8 -> 11.4 | Improved |
| add the audio stream (early fusion) to the multisensory network (ablation) | On/off separation model · VoxCeleb separation (all) | On/off L1 error: 11.6 -> 11.4 | Improved |
| use stereo instead of mono audio (ablation) | On/off separation model · VoxCeleb separation (all) | On/off L1 error: 11.4 -> 11.4 | No clear effect |
| add self-supervised pretraining versus scratch (ablation) | On/off separation model · VoxCeleb separation (all) | On/off L1 error: 12.9 -> 11.4 | Improved |
| add permutation invariant (PIT) loss variant | On/off separation model · VoxCeleb separation (all) | SDR: 7.0 -> 7.6 | Improved |
| use audio-visual model instead of audio-only u-net PIT | On/off + PIT · VoxCeleb separation (all) | SDR: 7.3 -> 7.6 | Improved |
| use ours instead of Hou et al. on short clips | Ours (on/off) · VoxCeleb short videos (200ms) | On-SDR: 4.5 -> 7.6 | Improved |
Closest papers by meaning
- Self-Supervised Learning of Audio-Visual Objects from Video (ECCV 2020 · similarity 0.90)
- Whence the Voice? Self-supervised Dual-source Audio-Visual Localisation via Selective Convergence (ECCV 2026 · similarity 0.90)
- Audio-Visual Contrastive Learning with Temporal Self-Supervision (AAAI 2023 · similarity 0.89)
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.89)
- Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling (CVPR 2024 · similarity 0.87)
- Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
