Research map / 3D vision / 3D generation
Twinner: Shining Light on Digital Twins in a Few Snaps
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Twinner is a feed-forward large reconstruction model that predicts shape, PBR materials, and environment illumination from a few posed images, using a memory-efficient tricolumn voxel representation, procedural synthetic PBR data, and real-data shading loss.
Problem
Recovering relightable digital twins (geometry, PBR materials, and scene illumination) from only a few posed casual images, which today requires slow per-scene optimization and suffers from scarce PBR ground truth and synthetic-to-real domain gap.
Approach
A tricolumn representation: triplane feature vectors where each vector stacks the corresponding voxel column, so token count is quadratic in voxel resolution while voxels stay explicit; combined with a DiT illumination predictor and fine-tuning on real CO3Dv2 videos via a differentiable PBR shader and background-as-environment supervision.
Key results
- On StanfordORB relighting, Twinner achieves PSNR-H 24.55 and LPIPS 0.038, outperforming MetalLRM (21.50 PSNR-H, 0.052 LPIPS) and approaching single-scene optimizers like Neural-PBIR (26.01) at seconds vs hours.
- Twinner reaches angular illumination error 8.09 vs 11.35 for DiffusionLight, a roughly 28% improvement over the best prior method.
- Tricolumn representation improves LPIPS by about 22% over triplane (0.029 vs 0.037) in a LightplaneLRM ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add procedurally generated ZeroVerse training data (ablation) | Twinner · StanfordORB, novel scene relighting | LPIPS: 0.048 -> 0.038 | Helped |
| add real-data shading-loss training (ablation) | Twinner · StanfordORB, novel scene relighting | LPIPS: 0.042 -> 0.038 | Helped |
| add procedurally generated ZeroVerse training data (ablation) | Twinner · StanfordORB, novel view synthesis | LPIPS: 0.048 -> 0.037 | Helped |
| add real-data shading-loss training (ablation) | Twinner · StanfordORB, novel view synthesis | LPIPS: 0.041 -> 0.037 | Helped |
| replace triplane with tricolumn representation (ablation) | LightplaneLRM · StanfordORB, novel view synthesis | LPIPS: 0.037 -> 0.029 | Helped |
| predict illumination from four input views with Twinner | Twinner · StanfordORB, illumination prediction | Angular Error: 11.35 -> 8.09 | Helped |
| use Twinner instead of MetalLRM | feed-forward reconstruction · StanfordORB, novel scene relighting | LPIPS: 0.052 -> 0.038 | Helped |
| use Twinner instead of MetalLRM | feed-forward reconstruction · StanfordORB, novel view synthesis | LPIPS: 0.051 -> 0.037 | Helped |
Surprising findings
- Twinner's novel-view synthesis PSNR-L on StanfordORB (29.62) is below some single-scene optimizers, and tricolumn has lower SSIM (0.966) than triplane (0.972) in the ablation.
Limitations
- Only four input views are used for feed-forward predictors, a disadvantage relative to single-scene optimizers that use all views.
- Assumes environment light is constant in space, so shadows cast by the object are not modeled.
- Synthetic procedural assets are not realistic, requiring real-data fine-tuning.
Experimental setup
Datasets: Objaverse-like artist-created assets (140k) Zeroverse (procedural, MatSynth materials, PolyHaven environment maps) CO3Dv2 StanfordORB MatSynth PolyHaven
Benchmarks: StanfordORB
Metrics: PSNR-H PSNR-L SSIM LPIPS Depth error Geometry error Shape Chamfer Angular error RMSE Normalized RMSE
Baselines: MetalLRM NVDiffRec NVDiffRecMC Neural-PBIR NeRFactor NeRD PhySG InvRender NMF TensoIR StyleLight DiffusionLight
Builds on: LightplaneLRM Large Reconstruction Model (LRM) Diffusion Transformer (DiT) MetalLRM Zeroverse
Closest papers by meaning
- Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Dataset (CVPR 2025 · similarity 0.87)
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.86)
- RelitLRM: Generative Relightable Radiance for Large Reconstruction Models (ICLR 2025 · similarity 0.86)
- HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces (CVPR 2024 · similarity 0.85)
- SplitNeRF: Split Sum Approximation Neural Field for Joint Geometry, Illumination, and Material Estimation (NeurIPS 2024 · similarity 0.85)
- Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.