Research map / Vision-language and multimodal / Multimodal LLMs

CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CLiViS, a training-free framework where an LLM plans sub-tasks for a VLM and iteratively updates a dynamic cognitive map and evidence memory to answer questions over egocentric video.

Problem

Embodied visual reasoning over long egocentric videos with complex free-form instructions; Socratic LLM-over-captions pipelines lose visual detail, while end-to-end VLMs struggle with stepwise compositional reasoning.

Approach

An LLM acts as high-level planner that issues sub-instructions to a VLM for focused perception on video segments, and the resulting evidence updates a structured cognitive map (navigation graph plus relation graph) and an evidence memory over multiple rounds, without any training.

Key results

• CLiViS with InternVL3 reaches 55.4% on OpenEQA, 23.0% on EgoTempo, and 69.4% on EgoSchema, average 49.3% across the three benchmarks, the best among compared methods.
• With Qwen2.5-VL as the VLM, CLiViS improves the baseline average from 40.6% to 44.9% (+4.3) across the three benchmarks.
• Collapsing the multi-round LLM-VLM interaction to a single round drops EgoTempo accuracy by 10.5%; replacing Qwen2.5-Max with InternVL3 as the high-level reasoner drops it by 12.4%.

Surprising findings

• Replacing the LLM planner with a VLM causes the largest ablation drop, suggesting a dedicated strong LLM is essential rather than supplementary.
• Gains over the base VLM are small for VideoLLaMA3 on OpenEQA (+0.2%), where the baseline is already strong.

Limitations

• Relies on closed-source API LLMs (Qwen2.5-Max, DeepSeek-V3) for high-level planning, limiting reproducibility and adding cost.
• Latency is still high (195 s per EgoSchema question) relative to the fastest training-free baseline.
• LLM-as-judge scoring with Qwen2.5-Max for open-ended benchmarks may introduce evaluation bias.

Experimental setup

Datasets: OpenEQA EgoTempo EgoSchema

Benchmarks: OpenEQA EgoTempo EgoSchema

Metrics: accuracy

Baselines: VideoAgent VideoTree LVNet Video-R1 AlanaVLM Qwen2.5-VL InternVL3 VideoLLaMA3

Builds on: VideoAgent VideoTree Socratic models

Closest papers by meaning

• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.89)
• UniVR: Thinking in Visual Space for Unified Visual Reasoning (NeurIPS 2026 · similarity 0.88)
• Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models (ICLR 2026 · similarity 0.88)
• PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments (NeurIPS 2025 · similarity 0.88)
• ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation (CVPR 2026 · similarity 0.88)
• BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception (NeurIPS 2025 · similarity 0.88)