Research map / 3D vision / Face and shape
GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Improves the Large Reconstruction Model (LRM) with a convolutional encoder, pixelshuffle upsampler, NeRF-to-mesh geometry refinement via differentiable marching cubes, and a 4-second per-instance texture refinement for feed-forward multi-view-to-3D mesh reconstruction.
Problem
Feed-forward 3D reconstruction from multi-view images via large reconstruction models often yields poor textures (e.g., text, portraits), grid artifacts, and degraded geometry when meshes are extracted with marching cubes from NeRF fields.
Approach
Replace the DINO encoder with a trained-from-scratch convolutional encoder and deconvolution with pixelshuffle, fine-tune the NeRF through differentiable mesh rendering (DiffMC) with depth/mask/normal losses, and add a per-instance texture refinement that fine-tunes the triplane and color MLP on input views while freezing the generator.
Key results
- On GSO, texture-refined GTR reaches PSNR 29.788, SSIM 0.960, LPIPS 0.047, versus 24.740 PSNR for InstantMesh (NeRF).
- On OmniObject3D, GTR with texture refinement reports PSNR 25.400 and LPIPS 0.059, with CD 0.504 and IoU 0.728 for the feed-forward model.
- Per-instance texture refinement takes about 4 seconds (20 steps on 4 views) on an A100, and mesh generation takes about 1 second.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add per-instance texture refinement (ablation) | GTR · GSO | PSNR: 28.673 -> 29.788 | Helped |
| Add per-instance texture refinement (ablation) | GTR · GSO | LPIPS: 0.055 -> 0.047 | Helped |
| Add per-instance texture refinement (ablation) | GTR · OmniObject3D | PSNR: 25.372 -> 25.400 | Helped |
| Add per-instance texture refinement (ablation) | GTR · OmniObject3D | LPIPS: 0.06 -> 0.059 | Helped |
| Replace NeRF+MC extraction with NeRF-initialized geometry refinement using mesh rendering (ablation) | GTR · GSO | image reconstruction quality | Helped |
| Add normal loss during geometry refinement (ablation) | GTR · 3D mesh surfaces | surface quality | Helped |
| Replace pre-trained DiNO ViT with a convolutional image encoder (ablation) | GTR · validation training | PSNR convergence | Mixed |
| Use a pretrained VAE encoder instead of the convolutional encoder trained from scratch (ablation) | GTR · training | early convergence | Helped |
| Train on 100k Objaverse assets only instead of the mixed dataset (ablation) | GTR · training | early performance | No clear effect |
Surprising findings
- Using a DINO ViT encoder did not show improved convergence early in training compared to the convolutional encoder.
- Extracting meshes with marching cubes from the NeRF degrades 2D rendering scores significantly, which geometry refinement recovers.
Limitations
- Triplane resolution of 256 can be insufficient for fine geometric details.
- Requires camera-conditioned (posed) multi-view input.
- Focused on object-centric assets, not complex scenes.
Experimental setup
Datasets: Objaverse internal 3D asset dataset GSO OmniObject3D
Benchmarks: GSO OmniObject3D
Metrics: PSNR SSIM LPIPS Chamfer Distance IoU
Baselines: LRM SV3D CRM LGM InstantMesh MeshLRM
Builds on: LRM InstantMesh Differentiable Marching Cubes Objaverse Zero123++
Closest papers by meaning
- LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields (CVPR 2025 · similarity 0.89)
- LRM: Large Reconstruction Model for Single Image to 3D (ICLR 2024 · similarity 0.89)
- LRM-Zero: Training Large Reconstruction Models with Synthesized Data (NeurIPS 2024 · similarity 0.88)
- NeRFPrior: Learning Neural Radiance Field as a Prior for Indoor Scene Reconstruction (CVPR 2025 · similarity 0.87)
- UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections (NeurIPS 2024 · similarity 0.87)
- G3DR: Generative 3D Reconstruction in ImageNet (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.