Research map / Video / Video segmentation and tracking
LoCoNet: Long-Short Context Network for Active Speaker Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoCoNet, an end-to-end parallel-inference active speaker detection network that interleaves long-term intra-speaker attention modeling with short-term inter-speaker convolution modeling, reaching SOTA on AVA-ActiveSpeaker, Talkies, and Ego4D.
Problem
Active speaker detection requires identifying who is speaking per frame; prior parallel-inference methods model long-term single-speaker context but ignore multi-speaker interactions, while sequential methods are computationally expensive.
Approach
Interleave Long-term Intra-speaker Modeling (self-attention for temporal dependencies plus audio-visual cross-attention) with Short-term Inter-speaker Modeling (small convolutions over nearby frames across speakers), trained end-to-end with multi-block supervision, plus a VGGFrame audio encoder that keeps per-frame temporal resolution using pretrained VGGish weights.
Key results
- Achieves 95.2% mAP on AVA-ActiveSpeaker, +0.3% over SPELL+ with about 38x fewer average-FLOPs.
- Achieves 97.2% mAP on Talkies (+2.7%) and 68.4% mAP on Ego4D (+7.7% over the challenge winner).
- On AVA with 3 visible faces, improves mAP by 3.0% over prior methods (87.4% vs 84.4% for light-ASD).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ResNet-34 with VGGFrame as the audio encoder (ablation) | LoCoNet · AVA-ActiveSpeaker | mAP: 91.2 -> 92.8 | Helped |
| Add 3 LIM modules (ablation) | LoCoNet with VGGFrame · AVA-ActiveSpeaker | mAP: 92.8 -> 94.1 | Helped |
| Add 3 SIM modules (ablation) | LoCoNet with VGGFrame · AVA-ActiveSpeaker | mAP: 92.8 -> 94.0 | Helped |
| Add both LIM and SIM modules (ablation) | LoCoNet with VGGFrame · AVA-ActiveSpeaker | mAP: 92.8 -> 95.2 | Helped |
| Increase input length from 20 to 100 frames (ablation) | LoCoNet with S=1 · AVA-ActiveSpeaker | mAP: 87.8 -> 92.8 | Helped |
| Increase input length from 100 to 200 frames (ablation) | LoCoNet with S=1 · AVA-ActiveSpeaker | mAP: 92.8 -> 93.2 | Helped |
| Increase the number of modeled speakers from 1 to 2 (ablation) | LoCoNet with T=200 · AVA-ActiveSpeaker | mAP: 93.2 -> 94.6 | Helped |
| Increase the number of modeled speakers from 2 to 3 (ablation) | LoCoNet with T=200 · AVA-ActiveSpeaker | mAP: 94.6 -> 95.2 | Helped |
| Increase SIM receptive field from 1 to 7 frames (ablation) | LoCoNet with S=3 and T=200 · AVA-ActiveSpeaker | mAP: 94.5 -> 95.2 | Helped |
| Increase SIM receptive field beyond 7 frames (ablation) | LoCoNet with S=3 and T=200 · AVA-ActiveSpeaker | mAP: 95.2 -> 95.2; 95.2 -> 95.0; 95.2 -> 94.9 | Mixed |
| Replace SIM convolution with window self-attention (ablation) | LoCoNet · AVA-ActiveSpeaker | mAP: 95.2 -> 94.4 | Hurt |
| Add LSCM blocks, increasing N from 0 to 1 (ablation) | LoCoNet · AVA-ActiveSpeaker | mAP: 92.8 -> 94.3 | Helped |
Surprising findings
- Long-term intra-speaker context gains are large but saturate; SIM receptive fields beyond 7 frames give no benefit and reduce mAP slightly.
Limitations
- Uses only 8 seconds of context; larger contexts not studied.
- Failure cases remain when the active speaker is far and less salient.
Experimental setup
Datasets: AVA-ActiveSpeaker Talkies Ego4D Audio-Visual benchmark AudioSet
Benchmarks: AVA-ActiveSpeaker Talkies Ego4D Audio-Visual benchmark
Metrics: mAP AUC average-FLOPs
Baselines: ASC MAAS ASDNet SPELL SPELL+ UniCon TalkNet ASD-Trans EASEE Light-ASD TS-TalkNet
Builds on: TalkNet VGGish Transformer
Closest papers by meaning
- LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale (CVPR 2025 · similarity 0.83)
- LoCoCo: Dropping In Convolutions for Long Context Compression (ICML 2024 · similarity 0.83)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.83)
- Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs (NeurIPS 2026 · similarity 0.83)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.82)
- SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models (CVPR 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.