Research map / 3D vision / 3D generation
Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Digital Twin Catalog, a dataset of 2,000 scanned, millimeter-accurate PBR 3D object models with DSLR and egocentric Aria captures, and benchmarks inverse rendering, novel view synthesis, and robotic pushing/grasping on it.
Problem
Existing real-world object datasets lack digital-twin-quality geometry and PBR materials, and there is no benchmark for evaluating object reconstruction from egocentric (AR glasses) imagery.
Approach
A large set of 2,000 industrially scanned objects with millimeter geometry and photorealistic PBR materials, paired with DSLR multi-lighting captures and Aria egocentric recordings aligned to the 3D objects via differentiable rendering, serving as a digital-twin-grade benchmark.
Key results
- On DSLR inverse rendering, NVDiffRecMC achieved the best shape Chamfer distance (1.34e-3) and relighting PSNR-H 27.78, while InvRender achieved the highest novel-scene relighting PSNR-H of 29.52 among those compared.
- On egocentric novel view synthesis, 3D-GS reached PSNR 28.81 and LPIPS 0.020, slightly above 2D-GS (PSNR 28.75).
- Robotic policies trained on DTC cups reached 47.0% pushing success at 5cm versus 40.3% for Objaverse-XL cups, and 42.7% vs 38.6% grasping success.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the policy using DTC objects instead of Objaverse-XL objects | Robotic pushing policy · Robotic pushing @ 2cm | Success rate: 25.3% ± 6.0% -> 36.3% ± 1.5% | Helped |
| Train the policy using DTC objects instead of Objaverse-XL objects | Robotic pushing policy · Robotic pushing @ 3cm | Success rate: 29.7% ± 6.0% -> 43.7% ± 1.2% | Helped |
| Train the policy using DTC objects instead of Objaverse-XL objects | Robotic pushing policy · Robotic pushing @ 5cm | Success rate: 40.3% ± 5.5% -> 47.0% ± 2.6% | Helped |
| Train the policy using DTC objects instead of Objaverse-XL objects | Robotic grasping policy · Robotic grasping | Success rate: 38.6% ± 11.0% -> 42.7% ± 4.7% | Helped |
Surprising findings
- Policies trained on DTC objects outperformed those trained on Objaverse-XL objects especially at stricter pushing thresholds (36.3% vs 25.3% at 2cm).
Limitations
- Creating digital twins requires deliberate hardware setup and substantial human refinement, especially for glossy materials.
- Hardware is limited to objects of a certain size and cannot recover deformable, highly specular, or transparent objects.
- Post-processing takes about 4 hours per object for 4K PBR maps, limiting scalability.
Experimental setup
Datasets: DTC Objaverse-XL Stanford-ORB
Benchmarks: DTC DSLR inverse rendering benchmark DTC egocentric novel view synthesis benchmark PyBullet pushing and grasping success rate
Metrics: PSNR SSIM LPIPS depth SI-MSE normal error shape Chamfer distance pushing success rate grasping success rate
Baselines: NVDiffRec NVDiffRecMC InvRender PhySG Neural-PIL NeRD 3D-GS 2D-GS
Builds on: 3D Gaussian splatting Neural-PBIR Stanford-ORB
Closest papers by meaning
- Twinner: Shining Light on Digital Twins in a Few Snaps (CVPR 2025 · similarity 0.87)
- ArticulatedGS: Self-supervised Digital Twin Modeling of Articulated Objects using 3D Gaussian Splatting (CVPR 2025 · similarity 0.83)
- SyncHuman: Synchronizing 2D and 3D Generative Models for Single-view Human Reconstruction (NeurIPS 2025 · similarity 0.83)
- DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans (CVPR 2024 · similarity 0.82)
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.82)
- Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.