Research map / Vision-language and multimodal / Multimodal LLMs

3D Space as a Scratchpad for Editable Text-to-Image Generation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a 3D spatial scratchpad in which an LLM agent pipeline parses a prompt, instantiates subjects as image-to-3D meshes, plans placement, orientation and camera in 3D, and renders the scene as depth plus identity conditioning for SIGMA-Gen image generation, enabling editable text-to-image generation without additional training.

Problem

Text-to-image generators and VLMs struggle with precise spatial relations, object identity consistency and compositional coherence for complex prompts, and 2D layout-based planning is limited for spatial reasoning.

Approach

Use an explicit editable 3D scene (meshes placed by LLM agents) as an intermediate reasoning workspace, rendered back to depth and identity-conditioned image generation, so 3D edits propagate to the final image while preserving subject identity.

Key results

• On GenAI-Bench, the method reaches 0.83 text alignment versus 0.63 for Flux.1 [dev] and 0.80 for Idea2Img, with image quality 4.81.
• On CompoundPrompts, text alignment is 0.91 versus 0.85 for Flux and 0.87 for Idea2Img.
• On T2I-CompBench, 3D spatial score is 0.54 versus 0.43 for RPG-Flux and 0.39 for Flux.

Surprising findings

• Providing the LLM with renders of the scene and asking for transformation matrices directly fails, and using full generated images for orientation estimation is unreliable.
• Adding rulers to scratchpad renders improves text alignment on GenAI-Bench.

Limitations

• LLM-generated 3D arrangements can be too uniform and not reflect real-world compositions.
• Generated meshes have limited pose diversity and articulated assets cannot be reposed.
• Activity and orientation understanding may fail in complex interactions.

Experimental setup

Datasets: GenAI-Bench CompoundPrompts T2I-CompBench

Benchmarks: GenAI-Bench CompoundPrompts T2I-CompBench

Metrics: VQAScore Q-Align UniDet-based spatial score 3-in-1 metric

Baselines: Flux.1 [dev] Idea2Img RPG - SDXL RPG - Flux Iterative Insert-Anything DALL-E-3

Builds on: SIGMA-Gen Idea2Img RPG Hunyuan-3D 2.5 Flux.1 [dev]

Closest papers by meaning

• Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.89)
• LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models (CVPR 2025 · similarity 0.89)
• Socratic-Geo: Synthetic Data Generation and Cross-Modal Geometric Reasoning via Multi-Agent Interaction (CVPR 2026 · similarity 0.88)
• Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing (NeurIPS 2025 · similarity 0.88)
• Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views (CVPR 2026 · similarity 0.88)
• Think, Plan, Paint: Layout-Aware Reasoning for Controllable Image Generation in Unified Models (NeurIPS 2026 · similarity 0.88)