Research map / Video / Video segmentation and tracking
Self-Supervised Learning of Audio-Visual Objects from Video
Triantafyllos Afouras, Andrew Owens, Joon Son Chung, Andrew Zisserman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add optical-flow aggregation of attention over time (ablation) | audio-visual object model · Talking head detection (LRS3) | detection accuracy: 94.2% -> 99.7% | Improved |
| add optical-flow aggregation of attention over time (ablation) | audio-visual object model · Talking head detection (LRS2) | detection accuracy: 98.4% -> 99.6% | Improved |
| enlarge NMS window to rho=150 in no-flow model (ablation) | no-flow variant · Talking head detection (LRS3) | detection accuracy: 94.2% -> 97.2% | Improved |
| add flow on high-motion videos (ablation) | audio-visual object model · Talking head detection (LRS3-test high motion) | detection accuracy: 88.1% -> 99.3% | Improved |
| add flow on high-motion videos (ablation) | audio-visual object model · Talking head detection (LRS2-val high motion) | detection accuracy: 93.6% -> 96.1% | Improved |
| end-to-end training instead of frozen pretrained features (ablation) | source separation · LRS2 (3 speakers) | SDR: 7.0 -> 7.2 | Improved |
| use rho=150 instead of rho=100 (ablation) | source separation · LRS2 (3 speakers) | WER %: 44.7 -> 42.0 | Improved |
| use face-detector tracks instead of audio-visual objects (ablation) | source separation · LRS2 (3 speakers) | WER %: 44.9 -> 42.0 | No clear effect |
| replace supervised face detection with self-supervised objects | source separation · LRS2 (2 speakers) | SDR: 11.3 -> 10.8 | Hurt |
| use self-supervised object embeddings instead of SyncNet | active speaker detection · Columbia | average F1: 89.5 -> 90.8 | Improved |
| use self-supervised objects instead of Perfect Match supervised | synchronization · LRS2 (5 input frames) | accuracy: 88.1 -> 78.8 | Hurt |
| use self-supervised model instead of RetinaFace MobileNet | non-human speaker localization · Simpsons single-head | localization accuracy: 72.1 -> 98.8 | Improved |
Closest papers by meaning
- Self-Supervised Object Detection From Audio-Visual Correspondence (CVPR 2022 · similarity 0.92)
- Language-Guided Audio-Visual Source Separation via Trimodal Consistency (CVPR 2023 · similarity 0.91)
- Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (ECCV 2018 · similarity 0.90)
- Look, Listen and Learn (ICCV 2017 · similarity 0.90)
- Self-Supervised Audio-Visual Soundscape Stylization (ECCV 2024 · similarity 0.90)
- Telling Left From Right: Learning Spatial Correspondence of Sight and Sound (CVPR 2020 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
