Research map / Vision-language and multimodal / Multimodal LLMs

EgoSound: Benchmarking Sound Understanding in Egocentric Videos

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EgoSound, a 7,315-QA open-ended benchmark over 900 Ego4D and EgoBlind videos for evaluating sound understanding in omni MLLMs, finding best model Qwen3-Omni-Thinking-30B reaches 56.7% vs 83.9% human.

Problem

Existing egocentric video QA benchmarks rely only on visual cues and ignore audio, so MLLMs' ability to understand sound in first-person videos is not systematically evaluated.

Approach

A multi-stage auto-generative pipeline (interaction annotation with Qwen2.5-VL, audio-visual captioning with Gemini-2.5, open-ended QA generation with GPT-4o) builds sound-centric QA over sighted (Ego4D) and sound-dependent (EgoBlind) egocentric videos across seven sound-centric task categories, scored by GPT-5 as judge.

Key results

• Best model Qwen3-Omni-Thinking-30B reaches 56.7% average accuracy versus 83.9% for human evaluators.
• Audio-only ablation on Qwen3-Omni-Thinking-30B drops Location by 28.1 points and Causality by 24.9 points.
• EgoGPT-7B, the egocentric-specialized model, averages 34.3%, below most comparable-scale omni models.

Surprising findings

• Egocentric-specialized EgoGPT-7B underperforms general omni models of comparable scale.
• Models often perform worse on sound characteristics and spatial localization than on counting or causality, despite strong visual perception.

Limitations

• Egocentric model comparison is limited by few open-source egocentric MLLMs, so EgoGPT conclusions are not robust.
• Benchmark QA generated by proprietary models (Gemini-2.5, GPT-4o) with GPT-5 as judge may introduce bias; Gemini excluded from evaluation for fairness.

Experimental setup

Datasets: Ego4D EgoBlind

Benchmarks: EgoSound

Metrics: Accuracy Score (0-5)

Baselines: VideoLLaMA2.1-AV-7B video-SALMONN 2+ 7B video-SALMONN 2+ 72B Qwen2.5-Omni-3B Qwen2.5-Omni-7B MiniCPM-o 2.6-8B Qwen3-Omni-Instruct-30B Qwen3-Omni-Thinking-30B EgoGPT-7B

Builds on: Ego4D EgoBlind EgoVQA EgoTaskQA

Closest papers by meaning

• Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization (NeurIPS 2025 · similarity 0.90)
• Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.89)
• Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues (CVPR 2025 · similarity 0.89)
• Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos (CVPR 2024 · similarity 0.89)
• SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos (CVPR 2024 · similarity 0.88)
• STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence (ICLR 2026 · similarity 0.88)