Research map / Vision-language and multimodal / Cross-modal learning
Cross-Modal Supervision for Learning Active Speaker Detection in Video
Punarjay Chakravarty, Tinne Tuytelaars
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add temporal-continuity weighting of samples to VAD-supervised person-specific training (ablation) | Person-specific active speaker classifier (VAD) · Masters (speaker-specific) | mean average AUC: 0.65 -> 0.82 | Improved |
| train person-specific classifiers with directional audio instead of a generic classifier | Active speaker classifier · Masters | average AUC: 0.69 -> 0.81 | Improved |
| use VAD with hard-max posterior and no temporal weighting for person-specific training instead of directional audio | Person-specific classifier · Masters (speaker-specific) | mean average AUC: 0.81 -> 0.65 | Hurt |
| weak supervision by VAD instead of full directional audio for the generic classifier | Generic active speaker classifier · Masters | average AUC: 0.69 -> 0.71 | No clear effect |
| add online adaptation with temporally weighted samples (up to 10 s per person) | Prior generic classifier · Columbia | AUC: about 5-15% improvement | Improved |
| add temporal smoothing of thresholded scores up to about 3 seconds | Online-learnt classifier · Columbia | F-score: plateauing of results at around 3 seconds | Improved |
Closest papers by meaning
- MAAS: Multi-Modal Assignation for Active Speaker Detection (ICCV 2021 · similarity 0.89)
- End-to-End Active Speaker Detection (ECCV 2022 · similarity 0.89)
- A Light Weight Model for Active Speaker Detection (CVPR 2023 · similarity 0.89)
- C3ASD: Multi-Level Consistency-Driven Representation Learning for Robust Active Speaker Detection (ECCV 2026 · similarity 0.88)
- How To Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild (ICCV 2021 · similarity 0.88)
- LoCoNet: Long-Short Context Network for Active Speaker Detection (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
