Research map / Vision-language and multimodal / Multimodal LLMs
Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces E3VQA, a 4K-question multiple-choice benchmark of synchronized ego-exo image pairs, and M3CoT, a training-free prompting method that fuses three multi-perspective scene graphs via iterative multi-agent refinement, improving GPT-4o and Gemini 2.0 Flash over CCoT.
Problem
Egocentric-only LVLM inputs have narrow field of view and lack global context, so models fail on spatially or contextually demanding queries; it is unclear whether LVLMs can jointly reason over synchronized ego and exo views.
Approach
Build three scene graphs per ego-exo pair (joint, ego-to-exo, exo-to-ego), iteratively refine them by exchanging information among three LVLM agents, and answer by majority vote over agents conditioned on the unified graphs.
Key results
- M3CoT reaches 68.58% average accuracy with GPT-4o vs 63.74% for CCoT (+4.84 points) on E3VQA.
- M3CoT reaches 66.12% with Gemini 2.0 Flash vs 60.18% for CCoT (+5.94 points).
- Best open-source model InternVL3-14B reaches 53.02% default accuracy, while GPT-4o achieves 60.90% default on E3VQA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply M3CoT instead of CCoT | GPT-4o · E3VQA | accuracy: +4.84% | Helped |
| Apply M3CoT instead of CCoT | Gemini 2.0 Flash · E3VQA | accuracy: +5.94% | Helped |
| Apply M3CoT instead of CCoT | GPT-4o · E3VQA, numerical reasoning | accuracy: +6.88% | Helped |
| Apply M3CoT instead of CCoT | Gemini 2.0 Flash · E3VQA, numerical reasoning | accuracy: +8.13% | Helped |
| Add scene-graph refinement (ablation) | M3CoT · E3VQA, 120-sample scene-graph subset | false discovery rate: 9.37% to 6.21% | Helped |
| Add scene-graph refinement (ablation) | M3CoT · E3VQA, 120-sample scene-graph subset | answer accuracy: 46.88% to 60.42% | Helped |
| Use only the Ego&Exo perspective rather than combining perspectives with refinement (ablation) | M3CoT · E3VQA, Both subset | accuracy: 50.87 -> 53.04 | Hurt |
| Use only the Ego2Exo perspective rather than combining perspectives with refinement (ablation) | M3CoT · E3VQA, Exo subset | accuracy: 61.51 -> 62.91 | Hurt |
| Use only the Exo2Ego perspective rather than combining perspectives with refinement (ablation) | M3CoT · E3VQA, Ego subset | accuracy: 68.02 -> 69.28 | Hurt |
| Provide both views instead of only the required single view (ablation) | LVLMs · E3VQA, Ego and Exo subsets | accuracy | Hurt |
| Provide both views instead of one view (ablation) | LVLMs · E3VQA, Both subset | accuracy | Helped |
Surprising findings
- Models consistently underperform on egocentric questions compared to exocentric questions.
- Most existing CoT methods yield marginal gains or degradation on open-source models, and CoT is reported as often ineffective on smaller models.
Limitations
- M3CoT incurs substantially higher token usage and latency than other CoT methods.
- Performance on the Both subset remains low (below 40% for single-view baselines and about 53% with M3CoT), so multi-view integration is still hard.
- Benchmark is multiple-choice with distractors built from model outputs and relies on GPT-4o for generation.
Experimental setup
Datasets: Ego-Exo4D LEMMA
Benchmarks: E3VQA LEMMA-based E3VQA
Metrics: accuracy false discovery rate
Baselines: GPT-4o GPT-4o mini Gemini 2.0 Flash Gemini 1.5 Pro Claude 3.5 Sonnet InternVL3-14B Qwen2.5-VL-7B DDCoT CoCoT CCoT
Builds on: Compositional Chain-of-Thought (CCoT) DDCoT CoCoT Ego-Exo4D
Closest papers by meaning
- LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences (CVPR 2025 · similarity 0.89)
- Multi-modal Situated Reasoning in 3D Scenes (NeurIPS 2024 · similarity 0.88)
- Hypo3D: Exploring Hypothetical Reasoning in 3D (ICML 2025 · similarity 0.88)
- Empowering Large Language Models with 3D Situation Awareness (CVPR 2025 · similarity 0.88)
- Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes (ICLR 2026 · similarity 0.88)
- Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.