Research map / Vision-language and multimodal / Multimodal LLMs
ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ORIGAMISPACE, a 350-instance origami benchmark with four MLLM spatial-reasoning tasks, an improved CP-code compiler with error feedback, and RL (TRICO/PPO) for CP code generation on Qwen2.5-VL-32B.
Problem
Existing spatial reasoning benchmarks rarely test multi-step spatial transformations under precise mathematical/geometric constraints, and rarely involve interactive physical manipulation. MLLMs' ability to generate valid folding programs under such constraints is unmeasured.
Approach
Use origami (crease pattern diagrams, compiled flat patterns, folding processes, folded shapes) as a verifiable multi-step spatial benchmark, with a compiler that returns structured error feedback (syntax, geometric impossibility, self-intersection, ambiguity) enabling interactive environmental learning and RL for CP code generation.
Key results
- Gemini2.5-pro achieves the best spatial relationship accuracy among evaluated models (e.g., 53.45% on multi-step spatial reasoning), well below human experts (100% on that task).
- Environmental learning with compiler feedback raises GPT-4o's CP code compilation pass rate to 66.92% from 28.56% in-context and its total score to 52.67 from 42.82.
- RL-trained Qwen2.5-VL-32B reaches a total code-generation score of 42.57, surpassing the untrained Qwen2.5-VL-72B in-context (32.68) and approaching the 72B model with environmental learning (40.58).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use environmental learning with iterative compiler feedback instead of in-context learning | Qwen2.5-VL-32B · CP Code Generation | Total: 30.19 -> 36.72 | Helped |
| Use environmental learning with iterative compiler feedback instead of in-context learning | Qwen2.5-VL-72B · CP Code Generation | Total: 32.68 -> 40.58 | Helped |
| Use environmental learning with iterative compiler feedback instead of in-context learning | GPT-4o · CP Code Generation | Total: 42.82 -> 52.67 | Helped |
| Use environmental learning with iterative compiler feedback instead of in-context learning | Gemini2.5-pro · CP Code Generation | Total: 42.80 -> 53.66 | Helped |
| Train with reinforcement learning using TRICO | Qwen2.5-VL-32B · CP Code Generation | Total: 36.72 -> 42.57 | Helped |
| Increase environmental-learning interaction rounds | MLLMs · CP Code Generation | compilation pass rate | Helped |
Surprising findings
- Top models perform close to random guessing (25%) on several tasks, with the weakest open models near chance on pattern prediction.
- Even with environmental learning, the best models satisfy only about 57% of mathematical constraints (constraint satisfaction score), indicating persistent failure on constraint adherence.
Limitations
- Benchmark is small (350 instances), with multiple-choice distractors designed manually.
- Performance saturates after 8-10 interaction rounds, suggesting inherent model bottlenecks remain.
- Constraint satisfaction remains low for top models even with interaction.
Experimental setup
Datasets: OrigamiSpace (350 instances) 471 CP-only data groups for RL training
Benchmarks: OrigamiSpace
Metrics: accuracy compilation pass rate topological structure similarity geometric similarity constraint satisfaction final folded state score
Baselines: Claude-3.5-Sonnet GPT-4o Gemini2.5-Flash Gemini2.5-pro Qwen2.5-VL-72B InternVL2.5-78B MiniCPM-o 2.6 NVILA-15B llava-1.5-7b VideoLLaMA3-7B deepseek-vl2
Builds on: PPO TRICO CLEVR LEGO-Puzzles
Closest papers by meaning
- EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs (CVPR 2026 · similarity 0.89)
- SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.89)
- MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.89)
- Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing (NeurIPS 2025 · similarity 0.89)
- SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models (CVPR 2025 · similarity 0.88)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.