Research map / 3D vision / 3D generation

PRISM: Programming Interactive Scenes from Monocular Images for Embodied Simulation

NeurIPS 2026 · 3D vision › 3D generation · approach: Monocular 3D object reconstruction

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes NeoWorld-Pro, which turns a single monocular image into executable Blender/URDF/USD scene programs via MLLMs and refines them with object-level and CEM-based scene-level physics-in-the-loop feedback in Isaac Sim.

Problem

Converting a single monocular image into simulation-ready interactive scenes with correct articulation, physical stability, and inter-object layout; existing image-to-URDF pipelines are open-loop and rely on curated asset libraries or category priors.

Approach

Reformulate monocular scene reconstruction as zero-shot MLLM procedural program synthesis (Blender scripts compiled to URDF/USD), with a two-level physics-in-the-loop refinement: an object-level critic loop for articulation and a scene-level CEM optimizer whose reward combines simulator penetration/drift penalties and an MLLM semantic score.

Key results

• Scene-level: NeoWorld-Pro achieves 0.00% scene and part penetration and 100% stability on the full 30-scene benchmark, versus 6.67% penetration for SAGE and 56.67% for VIGA.
• Object-level: NeoWorld-Pro recovers 216 of 245 ground-truth joints with 11.83% miss rate, versus 22.45% for Articraft and 61.22% for Articulate-Anything.
• Ablation: full method reaches 92.85% task success vs 44.05% without simulation feedback (open-loop) and 48.81% without the semantic reward.

Surprising findings

• Removing the MLLM semantic score drops task success to 48.81%, indicating physics rewards alone are insufficient for layout correctness.

Limitations

• Limited to rigid and articulated objects under quasi-static physics; no deformable materials, cloth, fluids.
• Evaluation limited to scenes with fewer than 10 objects.
• Relies on proprietary MLLMs and an MLLM judge for semantic scoring and articulation evaluation.

Experimental setup

Datasets: synthetic scene dataset (90 articulated categories, 30 USD scenes)

Benchmarks: synthetic scene benchmark (30 scenes) tabletop subset (11 scenes) articulated object benchmark (245 joints)

Metrics: SSIM LPIPS CLIP Uni3D Chamfer distance F-score joint miss rate joint type error axis error pivot error

Baselines: Articulate-Anything PhysX-Anything URDF-Anything+ Articraft TabletopGen VIGA SAGE

Builds on: NeoWorld Articulate-Anything Articraft Cross-Entropy Method

Closest papers by meaning

• Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image (NeurIPS 2026 · similarity 0.88)
• WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes (NeurIPS 2026 · similarity 0.86)
• Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model (ICLR 2025 · similarity 0.86)
• SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.86)
• PhysGen3D: Crafting a Miniature Interactive World from a Single Image (CVPR 2025 · similarity 0.85)
• PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI (CVPR 2024 · similarity 0.85)