Research map / Vision-language and multimodal / Multimodal LLMs

MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MetaSpatial, an RL framework that trains Qwen2.5-VL 3B/7B to generate 3D indoor scene layouts with format, physics, and GPT-4o rendering rewards, using a multi-turn refinement pipeline and the 3D-SPO advantage modulation.

Problem

VLMs lack internalized 3D spatial reasoning, so layout generation needs heavy post-processing, and SFT on single annotated layouts fails because valid layouts are a distribution without a unique ground truth.

Approach

A GRPO-style policy optimization (3D-SPO) that masks coordinate tokens and scales advantages by object-level collision and constraint penalties, combined with discounted multi-turn refinement trajectories and a composite format/physics/render reward.

Key results

• Qwen-VL 7B + MetaSpatial raises format accuracy from 0.85 to 0.98 and cuts collision rate from 38.2% to 11.5%.
• On Open3DVQA, MetaSpatial-7B reaches 73.5% qualitative and 35.6 quantitative average, above Qwen-VL-7B-SFT (72.8 / 34.1).
• 3D-SPO with T=5 outperforms GRPO at the same depth (collision 11.5% vs 13.7%, constraint 70.8% vs 76.2%).

Surprising findings

• Pure SFT on high-reward self-generated layouts improved format accuracy but lowered GPT-4o score (0.52 to 0.42) and raised collision rate.
• Performance dropped at T=7 relative to T=5 for multi-turn refinement, which the authors attribute to over-adjustment.

Limitations

• Constraint violation ratio remains high (70.8% even for the best model), and collision rates remain substantial.
• Dataset is limited to single-room, static-lighting indoor scenes; multi-room extension is future work.
• Rendering reward depends on GPT-4o and slow Blender rendering, limiting scalability.

Experimental setup

Datasets: Open3DVQA

Benchmarks: MetaSpatial layout test (format, GPT-4o score, collision, constraint) Open3DVQA

Metrics: format accuracy GPT-4o score collision ratio constraint violation ratio success rate absolute relative error

Baselines: GPT-4o LayoutGPT I-Design PPO GRPO Qwen-VL-7B-SFT LLaVA-7B

Builds on: GRPO PPO I-Design LayoutVLM Open3DVQA

Closest papers by meaning

• MindJourney: Test-Time Scaling with World Models for Spatial Reasoning (NeurIPS 2025 · similarity 0.90)
• LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models (CVPR 2025 · similarity 0.89)
• PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning (CVPR 2026 · similarity 0.89)
• S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance (CVPR 2026 · similarity 0.89)
• Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.89)
• SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning (CVPR 2026 · similarity 0.89)