Research map / Vision-language and multimodal / Multimodal LLMs

ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Rebuilds the VSI-Bench spatial reasoning benchmark with manually re-annotated 3D objects/geometry, regenerated debiased QAs, frame-budgeted variants (16/32/64/all), and visibility-based dummy-video stress tests to expose validity issues in VLM evaluation.

Problem

Existing video-based VLM spatial intelligence benchmarks derive QA from point-cloud 3D annotations that contain errors, and assume full-scene observability although VLMs see only sparsely sampled frames, so many QAs are incorrect or unanswerable.

Approach

Re-annotate objects, 3D boxes and room polygons directly against raw videos, regenerate QA with bias control and human verification, build nested frame-budget subsets with frame-aware answers, and use object-removed dummy videos to test whether models rely on visual evidence or priors.

Key results

• Open-source models drop substantially on ReVSI (up to 40%) on object counting, relative distance and direction, while proprietary models hold stable or improve; e.g., GPT-5.2 averages 50.9 on ReVSI vs 49.2 on VSI-Bench.
• Fine-tuned specialized 3D VLMs show smaller gains over base models on ReVSI than on VSI-Bench, and SpaceR-7B degrades on several tasks relative to its base model.
• On object-removed dummy videos for counting, InternVL3.5-8B scores 14.7 on Query-Drop and 17.7 on black videos while Qwen3-VL-8B scores 34.7 and 99.8, revealing prior-driven hallucination in InternVL3.5; fine-tuned models score near zero accuracy on counting dummy videos.

Surprising findings

• Proprietary models are under-assessed by VSI-Bench, e.g., on object counting, and open-source models' apparent advantage there stems from hallucination not penalized by biased data.
• Scaling Spatial-MLLM training data from 135k to 820k yields only marginal (~3%) gains on ReVSI.

Limitations

• Expert manual annotation is costly and limits scalability to training-scale supervision.
• Benchmark covers indoor scenes from five scan datasets only.
• Visibility threshold and GPT-5.2 name verification rely on partly automated steps.

Experimental setup

Datasets: ScanNetv2 ScanNet++ ARKitScenes 3RScan MultiScan VSI-Bench

Benchmarks: ReVSI VSI-Bench

Metrics: MRA exact-match accuracy

Baselines: VSI-Bench GPT-5.2 Gemini 3 Flash Gemini 3 Pro Qwen3-VL-8B-Instruct Qwen3-VL-32B-Instruct InternVL3.5-8B InternVL3.5-38B LLaVA-Video-7B-Qwen2 LLaVA-Video-72B-Qwen2 Cambrian-S-7B VST-7B-SFT

Builds on: VSI-Bench ScanNet ScanNet++ ARKitScenes 3RScan MultiScan

Closest papers by meaning

• MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.91)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.91)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.91)
• Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces (ICML 2026 · similarity 0.91)
• Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes (ICLR 2026 · similarity 0.90)
• InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)