Research map / 3D vision / 3D generation
MindJourney: Test-Time Scaling with World Models for Spatial Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Couples a frozen VLM with a pose-conditioned video diffusion world model (SWM or Stable-Virtual-Camera) via a beam search over camera trajectories, letting the VLM gather imagined views before answering spatial questions on SAT.
Problem
VLMs struggle to anticipate how a scene looks after egocentric motion, lacking an internal model of 3D dynamics needed for spatial reasoning.
Approach
Training-free test-time scaling: the VLM plans short camera trajectories, a controllable video world model renders the corresponding views, and the VLM scores and buffers helpful views (Spatial Beam Search) before answering with the gathered multi-view evidence.
Key results
- Average accuracy on SAT-Real rises 7.7 points on average across four VLM backbones; GPT-4o gains over 10 points (60.3 to 70.6 with SWM).
- o1 with MindJourney (SWM) reaches 84.7 on SAT-Real vs 74.6 for vanilla o1, and GPT-4.1 with SWM (80.6) surpasses vanilla o1.
- On SAT-Synthesized (500-question subset), mean accuracy improves about 8 points on average; o1 with SVC reaches 78.6 vs 72.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MindJourney with SWM | GPT-4o · SAT-Real | accuracy: 60.3 -> 70.6 | Helped |
| Add MindJourney with SWM | GPT-4.1 · SAT-Real | accuracy: 74.0 -> 80.6 | Helped |
| Add MindJourney with SWM | InternVL3-14B · SAT-Real | accuracy: 59.3 -> 66.6 | Helped |
| Add MindJourney with SWM | OpenAI o1 · SAT-Real | accuracy: 74.6 -> 84.7 | Helped |
| Add MindJourney with SWM | GPT-4o · SAT-Synthesized | accuracy: 61.0 -> 70.8 | Helped |
| Add MindJourney with SWM | GPT-4.1 · SAT-Synthesized | accuracy: 66.4 -> 75.4 | Helped |
| Add MindJourney with SWM | InternVL3-14B · SAT-Synthesized | accuracy: 61.4 -> 66.6 | Mixed |
| Add MindJourney to OpenAI o1 with SVC | OpenAI o1 · SAT-Synthesized | accuracy: 72.4 -> 78.6 | Helped |
| Remove trajectory descriptions (ablation) | MindJourney with GPT-4o and SWM · SAT-Real | accuracy: 68.0 -> 66.8 | Hurt |
| Remove trajectory descriptions (ablation) | MindJourney with GPT-4o and SWM · SAT-Synthesized | accuracy: 70.8 -> 64.1 | Hurt |
| Use 2 search steps and 20 video-generation iterations instead of 3 steps and 50 iterations (ablation) | MindJourney with GPT-4o and SWM · SAT-Real (time-consumption experiment) | accuracy: 70.6 -> 68.6 | Hurt |
| Increase video-generation iterations from 20 to 50 at 3 search steps (ablation) | MindJourney with GPT-4o and SWM · SAT-Real (time-consumption experiment) | accuracy: 69.3 -> 70.6 | Helped |
Surprising findings
- Plain VLM with world-model search (e.g., GPT-4.1 + SWM at 80.6) surpasses RL-fine-tuned o1 on SAT-Real.
- Performance on SAT-Real peaks at search depth 2 and drops at depth 3, attributed to world model degradation on outdoor/far-from-reference views.
Limitations
- Assumes a single reference view; multi-image queries are not handled.
- World model is question-agnostic and can hallucinate irrelevant or contradictory views; degrades on out-of-domain scenes, humans and animals, and with longer rollouts.
- Final accuracy depends on the VLM's own QA and scoring ability.
Experimental setup
Datasets: SAT HM3D RealEstate10K DL3DV-10K AI2-THOR
Benchmarks: SAT-Real SAT-Synthesized
Metrics: accuracy
Baselines: GPT-4o GPT-4.1 InternVL3-14B OpenAI o1 Stable-Virtual-Camera
Builds on: ReCamMaster Wan2.2-TI2V-5B Stable-Virtual-Camera Habitat 2.0
Closest papers by meaning
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.90)
- Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators (NeurIPS 2026 · similarity 0.90)
- Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations (ICLR 2026 · similarity 0.90)
- MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse (ICLR 2026 · similarity 0.90)
- SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.