Research map / 3D vision / 3D generation
PhysGen3D: Crafting a Miniature Interactive World from a Single Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free framework that turns a single image into an amodal, camera-centric 3D scene via pretrained vision models, then simulates it with MPM (Taichi Elements) and renders the result with physics-based rendering.
Problem
Generating physically plausible, controllable future video from a single static image, which current image-to-video diffusion models lack because they lack physical grounding and precise control.
Approach
Reconstructs an image-centric digital twin from one photo by chaining pretrained vision models (GPT-4o, Grounded-SAM, InstantMesh, DUSt3R, DiffusionLight, LaMa) with differentiable pose alignment, then simulates with a material point method engine and re-renders with shadow-mapped PBR compositing onto the inpainted background.
Key results
- Human evaluation (31 participants, 27 videos): PhysGen3D scores 3.707 on physical realism vs 2.811 for Kling 1.0 and 2.283 for Gen-3.
- GPT-4o evaluation: PhysGen3D reaches 0.752 physical realism and 0.796 alignment, versus 0.563 and 0.596 for Kling 1.0.
- On photorealism the method (3.411 human) trails Kling 1.0 (3.566) and Gen-3 (3.582), so rendering quality is comparable rather than superior.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PhysGen3D instead of Kling 1.0 | PhysGen3D · Human evaluation | PhysReal: 2.811 -> 3.707 | Helped |
| use PhysGen3D instead of Runway Gen-3 | PhysGen3D · Human evaluation | PhotoReal: 3.582 -> 3.411 | Hurt |
| use PhysGen3D instead of Kling 1.0 | PhysGen3D · Human evaluation | Align: 2.467 -> 3.866 | Helped |
| add VEnhancer post-processing (ablation) | PhysGen3D · VBench and GPT-4o evaluation | Imaging: 0.666 -> 0.677 | Helped |
| add VEnhancer post-processing (ablation) | PhysGen3D · VBench and GPT-4o evaluation | PhotoReal: 0.867 -> 0.880 | Helped |
| add VEnhancer post-processing (ablation) | PhysGen3D · VBench and GPT-4o evaluation | Align: 0.796 -> 0.745 | Hurt |
| remove pose optimization (ablation) | PhysGen3D · input-scene reconstruction | scene alignment | Hurt |
| remove inverse texture optimization (ablation) | PhysGen3D · input-scene rendering | color and brightness match | Hurt |
| remove point sampling (ablation) | PhysGen3D · MPM simulation | simulation completion | Hurt |
Surprising findings
- Gen-3 scores lowest on physical realism (2.283 human, 0.141 GPT-4o) despite the highest human photorealism among baselines.
- Without sampling points, MPM simulation crashes due to particle crowding in grid cells.
Limitations
- Designed for object-centric scenes with simple geometry; complex multi-object or occluded scenes are not handled.
- Rendering errors under extreme lighting and heavy shading, and penetration artifacts (e.g., pillow into ground).
- MPM struggles with fine thin structures such as kettle handles.
Experimental setup
Benchmarks: VBench
Metrics: physical realism photorealism semantic consistency motion smoothness imaging quality
Baselines: Kling 1.0 Runway Gen-3 Pika 1.5 MOFA-Video DragAnything
Builds on: InstantMesh Taichi Elements Grounded-SAM DUSt3R Mitsuba3
Closest papers by meaning
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.89)
- PhysX-3D: Physical-Grounded 3D Asset Generation (NeurIPS 2025 · similarity 0.87)
- RoboScape: Physics-informed Embodied World Model (NeurIPS 2025 · similarity 0.87)
- HoloScene: Simulation‑Ready Interactive 3D Worlds from a Single Video (NeurIPS 2025 · similarity 0.87)
- VideoPhy: Evaluating Physical Commonsense for Video Generation (ICLR 2025 · similarity 0.86)
- How Far Is Video Generation from World Model: A Physical Law Perspective (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.