Research map / Video / Video segmentation and tracking
The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Ego-Exocentric Conversational Graph Prediction task and AV-CONV, a self-attention model over time, subjects and modalities that jointly predicts speaking/listening edges among all participants from egocentric audio-visual video.
Problem
Inferring exocentric conversational interactions (who speaks to / listens to whom among all social partners) from egocentric videos, beyond behaviors directly involving the camera wearer.
Approach
Builds a dynamic directed conversational graph combining egocentric and exocentric edges, and uses a Conversational Attention module (cross-time, cross-subject, global-local self-attention) over per-head audio-visual tokens, with pairwise feature fusion for exocentric edges.
Key results
- AV-CONV reaches 86.15% average accuracy on egocentric edge predictions and 81.04% on exocentric edge predictions.
- AV-CONV outperforms the SAAL-based baseline by about 4.45% on egocentric tasks and the ASL+Layout baseline by a large margin.
- Removing cross-time attention, cross-subject attention, or global-local attention each reduces performance relative to the full model (DIRECT CONCAT is worst at 76.43 exo average vs 81.04).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-time attention (ablation) | AV-CONV · Egocentric Graph | accuracy: 82.00 -> 86.15 | Helped |
| add cross-subject attention (ablation) | AV-CONV · Egocentric Graph | accuracy: 82.00 -> 82.72 | Helped |
| add Global-Local attention (ablation) | AV-CONV · Egocentric Graph | accuracy: 82.00 -> 82.65 | Helped |
| add cross-time and cross-subject attention (ablation) | AV-CONV · Egocentric Graph | accuracy: 82.00 -> 85.14 | Helped |
| add cross-time and Global-Local attention (ablation) | AV-CONV · Egocentric Graph | accuracy: 82.00 -> 84.84 | Helped |
| add cross-subject and Global-Local attention (ablation) | AV-CONV · Egocentric Graph | accuracy: 82.00 -> 85.07 | Helped |
| add conversational attention (ablation) | AV-CONV · Egocentric Graph | accuracy: 82.00 -> 86.15 | Helped |
| use all three attention types rather than cross-time attention only (ablation) | AV-CONV · Egocentric Graph | accuracy: 84.35 -> 86.15 | Helped |
| use all three attention types rather than cross-subject attention only (ablation) | AV-CONV · Egocentric Graph | accuracy: 82.72 -> 86.15 | Helped |
| use all three attention types rather than Global-Local attention only (ablation) | AV-CONV · Egocentric Graph | accuracy: 82.65 -> 86.15 | Helped |
| use all three attention types rather than cross-time and cross-subject attention (ablation) | AV-CONV · Egocentric Graph | accuracy: 85.14 -> 86.15 | Helped |
| use all three attention types rather than cross-time and Global-Local attention (ablation) | AV-CONV · Egocentric Graph | accuracy: 84.84 -> 86.15 | Helped |
Surprising findings
- Audio alone performs well on the egocentric speaking task (88.57%) but fails on exocentric and listening tasks.
- Positional mask alone performs similarly to head-image-only input, yet beats head-only on listening sub-tasks.
Limitations
- Failure when multiple nearby subjects speak simultaneously, making speaker identification hard.
- Multi-channel audio provides limited high-resolution spatial information.
- Exocentric labels are derived by synchronizing egocentric annotations across camera wearers rather than directly annotated.
Experimental setup
Datasets: Egocentric Concurrent Conversations Dataset
Benchmarks: Egocentric Concurrent Conversations Dataset
Metrics: accuracy mAP
Baselines: SAAL ASL+Layout DIRECT CONCAT HEAD ONLY AUDIO ONLY MASK ONLY
Builds on: SAAL (Selective Auditory Attention Localization) Active Speaker Localization (ASL) TimeSformer-style space-time attention
Closest papers by meaning
- Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues (CVPR 2025 · similarity 0.88)
- ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction (CVPR 2025 · similarity 0.87)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.87)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.87)
- Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos (CVPR 2024 · similarity 0.87)
- Multi-speaker Attention Alignment for Multimodal Social Interaction (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.