Research map / 3D vision / 3D generation
LiteReality: Graphic-Ready 3D Scene Reconstruction from RGB-D Scans
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts RGB-D room scans into graphics-ready 3D scenes by retrieving artist-crafted CAD assets with a training-free retrieval pipeline, painting PBR materials from images, and assembling physics-enabled scenes in Blender.
Problem
Reconstructing real indoor RGB-D scans into compact, object-centric, interactive 3D replicas with PBR materials and physical properties, which existing reconstruction and simulation-scene generation methods do not deliver under clutter, occlusion, and poor lighting.
Approach
Combine scene-graph-based layout parsing with training-free hierarchical CAD retrieval (semantic filter, DINOv2 image matching, pose-aware rendering, MLLM contextual selection), then Material Painting via MLLM-guided material search and a lightweight LAB albedo shift, and procedural Blender assembly with MLLM-estimated masses.
Key results
- Training-free retrieval achieves avg/CAD Chamfer Distance 0.0986 vs 0.1042 for ScanNotate (best of baselines) on Scan2CAD ScanNet validation.
- Object-centric material painting reaches SSIM 0.4353 and LPIPS 0.5854, best SSIM/LPIPS among compared methods, with RMSE 0.2163 close to the best MIR+AO (0.2156).
- Full-scene graphics-ready reconstruction achieves RMSE 0.2664, SSIM 0.5818, LPIPS 0.6522, outperforming Phone2Proc and Digital Cousin variants on all metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LiteReality's training-free retrieval pipeline instead of ScanNotate | Object retrieval · ScanNet validation | avg/CAD L1 Chamfer Distance: 0.1042 -> 0.0986 | Helped |
| Use LiteReality material painting instead of MIR | Object-centric PBR material estimation · Object-centric PBR material benchmark | SSIM: 0.3981 -> 0.4353 | Helped |
| Use LiteReality material painting instead of MIR + AO | Object-centric PBR material estimation · Object-centric PBR material benchmark | RMSE: 0.2156 -> 0.2163 | Hurt |
| Add albedo-only optimization to MIR (ablation) | MIR material painting · Object-centric PBR material benchmark | RMSE: 0.2377 -> 0.2156 | Helped |
| Add albedo-only optimization to MIR (ablation) | MIR material painting · Object-centric PBR material benchmark | SSIM: 0.3981 -> 0.4203 | Helped |
| Add albedo-only optimization to MIR (ablation) | MIR material painting · Object-centric PBR material benchmark | LPIPS: 0.6111 -> 0.5899 | Helped |
| Use LiteReality instead of Digital Cousin | Full-scene graphics-ready reconstruction · Full-scene graphics-ready reconstruction | SSIM: 0.5531 -> 0.5818 | Helped |
Surprising findings
- Make-It-Real with the proposed albedo-only optimization achieves lowest RMSE on material painting, slightly beating the full LiteReality pipeline.
Limitations
- Reconstruction quality depends on upstream perception errors in detection and layout estimation.
- Object retrieval mismatches can noticeably affect scene realism.
- Constraint-based collision solver may leave interpenetrations in densely packed scenes.
Experimental setup
Datasets: ScanNet Scan2CAD ShapeNet 3D-FUTURE AI2-THOR Sketchfab
Benchmarks: Scan2CAD ScanNet LiteReality object-centric material benchmark LiteReality graphics-ready reconstruction benchmark
Metrics: L1 Chamfer Distance RMSE SSIM LPIPS
Baselines: MSCD Digital Cousin ScanNotate PhotoShape Make-It-Real Phone2Proc
Builds on: Scan2CAD Make-It-Real Digital Cousin Phone2Proc PhotoShape
Closest papers by meaning
- DRAWER: Digital Reconstruction and Articulation With Environment Realism (CVPR 2025 · similarity 0.88)
- HoloScene: Simulation‑Ready Interactive 3D Worlds from a Single Video (NeurIPS 2025 · similarity 0.86)
- ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary (CVPR 2025 · similarity 0.86)
- GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer (NeurIPS 2025 · similarity 0.85)
- Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials (NeurIPS 2024 · similarity 0.85)
- ARM: Appearance Reconstruction Model for Relightable 3D Generation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.