Research map / 3D vision / 3D generation

WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes

NeurIPS 2026 · 3D vision › 3D generation · approach: Generative indoor scene synthesis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Converts monolithic Marble-generated 3DGS worlds into object-decomposed, collision-ready scenes using an agent-guided SAM3 segmentation, 3DGS object extraction, background inpainting, and SAM3D object regeneration with pose alignment.

Problem

Generated 3D worlds (e.g., Marble) are static, monolithic 3DGS assets whose objects cannot be individually selected, edited, or physically interacted with, limiting embodied simulation and content creation.

Approach

An agent-looped pipeline that discovers interactable objects with a VLM, segments them across views with SAM3, lifts masks to Gaussian subsets via soft-assignment inverse rendering, inpaints and re-lifts the background, and replaces noisy extracted objects with SAM3D-generated meshes aligned by ICP and differentiable rendering.

Key results

• Interactable object recall rises from 25.40% (w/o agent) to 83.98% on MWM-easy, and 78.80% on the full MWM set.
• After object re-insertion, MANIQA improves from 0.3338 to 0.3621 while ReMOVE stays about the same (0.7933 vs 0.7934).
• Human MOS for object-level quality rises from 2.49 to 3.75 (overall quality) when converting original objects into new object assets, while scene-level scores drop slightly (4.41 to 4.13).

Surprising findings

• Scene-level MOS slightly decreases after conversion (overall quality 4.41 to 4.13 by humans), though object-level quality improves substantially.
• Ours w/o agent recall (23.77% overall) is far below the agent-guided version, showing VLM discovery is the main driver of coverage.

Limitations

• Depends on quality of the input 3D world model (Marble).
• Does not handle dynamic scenes, articulated objects, or physical properties such as mass and friction.
• Evaluation uses only six indoor scenes and no ground-truth decomposed objects or inpainted backgrounds.

Experimental setup

Datasets: Marble-World-Model (MWM) dataset

Benchmarks: MWM MWM-easy MWM-hard

Metrics: Interactable Object Recall ReMOVE MANIQA MOS (1-5 Likert)

Baselines: Ours w/o Agent Gaussian Grouping Marble (original scenes)

Builds on: 3D Gaussian Splatting SA3D SAM3 SAM3D DiffuEraser Marble

Closest papers by meaning

• WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis (NeurIPS 2026 · similarity 0.88)
• SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.87)
• PRISM: Programming Interactive Scenes from Monocular Images for Embodied Simulation (NeurIPS 2026 · similarity 0.86)
• Interactive3D: Create What You Want by Interactive 3D Generation (CVPR 2024 · similarity 0.86)
• 4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models (CVPR 2026 · similarity 0.86)
• WonderWorld: Interactive 3D Scene Generation from a Single Image (CVPR 2025 · similarity 0.86)