Research map / 3D vision / Face and shape

METASCENES: Towards Automated Replica Creation for Real-world 3D Scans

CVPR 2025 · 3D vision › Face and shape · approach: Generative textures for 3D humans · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds METASCENES, a 15k-object simulatable 3D scene dataset made by replacing ScanNet scanned objects with annotated Objaverse/generated assets, plus the SCAN2SIM multi-modal asset retrieval model and two benchmarks (micro-scene synthesis, cross-domain VLN).

Problem

Creating high-quality, diverse, interactable simulatable 3D scenes for embodied AI currently relies on artist-driven design, which does not scale; real-to-sim replacement of scanned objects lacks principled candidate selection.

Approach

Collect multiple asset candidates per scanned object (retrieval plus image/text-to-3D generation), have human annotators rank them to form ground-truth preference labels, and train a multi-modal contrastive model (SCAN2SIM) to select the best asset and align its pose using scanned point clouds.

Key results

• SCAN2SIM achieves 28.4% Top-1 and 76.0% Top-5 asset selection accuracy, beating GPT-4V (16.5% Top-1) and CLIP (14.9%).
• Training on METASCENES yields 52.64% SR on held-out scenes vs 51.21% for ProcTHOR, and 50.67% vs 45.33% on held-out ScanNet++ domains.
• Micro-scene synthesis benchmark reports PhyScene reduces object collision rate to 0.395 vs 0.645 for ATISS.

Measured effects reported in the paper

Surprising findings

• Image-only matching methods underperform point-cloud-based matching, and GPT-4V performs as second-best among baselines despite no training on the dataset.
• Training on both ProcTHOR and METASCENES did not beat METASCENES alone on held-out domains (46.67% vs 50.67% SR).

Limitations

• Asset selection accuracy remains modest (Top-1 28.4%), so automated replacement is far from solved.
• Scene graph relation accuracy was verified only on 10 randomly sampled scenes.

Experimental setup

Datasets: ScanNet Objaverse 3D-FRONT ProcTHOR ScanNet++

Benchmarks: METASCENES test set (optimal asset selection) ScanNet++ pose alignment Micro-Scene Synthesis cross-domain VLN (AI Habitat)

Metrics: Top-1 accuracy Top-5 accuracy Chamfer Distance Enhanced Chamfer Distance IoU Color Histogram FID SCA CKL collision rate

Baselines: ACDC ULIP-2 CLIP GPT-4V Uni3D Point-BERT PointNet++ ICP SSIM LPIPS ATISS DiffuScene

Builds on: Scan2CAD ULIP-2 ACDC SPOC ATISS DiffuScene PhyScene

Closest papers by meaning

• LiteReality: Graphic-Ready 3D Scene Reconstruction from RGB-D Scans (NeurIPS 2025 · similarity 0.80)
• PartGen: Part-level 3D Generation and Reconstruction with Multi-view Diffusion Models (CVPR 2025 · similarity 0.79)
• Muses: 3D-Controllable Image Generation via Multi-Modal Agent Collaboration (AAAI 2025 · similarity 0.79)
• CAT3D: Create Anything in 3D with Multi-View Diffusion Models (NeurIPS 2024 · similarity 0.79)
• Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabrication (NeurIPS 2024 · similarity 0.78)
• AutoPartGen: Autoregressive 3D Part Generation and Discovery (NeurIPS 2025 · similarity 0.78)