Research map / 3D vision / Face and shape
ARM: Appearance Reconstruction Model for Relightable 3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ARM, a feed-forward sparse-view image-to-3D model that reconstructs meshes with PBR textures by decoupling geometry (GeoRM triplane LRM) from appearance, with InstantAlbedo operating in UV texture space and a DINO-based material prior.
Problem
Image-to-3D feed-forward reconstruction models produce blurry textures and only per-vertex baked-in colors, failing to yield realistic, relightable appearance (spatially varying roughness/metalness and separated illumination).
Approach
Decouple geometry from appearance: predict shape with a triplane LRM, then back-project multi-view images, material-aware encodings and auxiliary data into UV texture space and decode albedo/lighting-baked color with a U-Net plus global-receptive FFC-Net; add a DINO-initialized material prior to disentangle material from illumination, and predict roughness/metalness with a separate GlossyRM.
Key results
- On GSO, ARM achieves F-score 0.968 and CD 0.049 vs. MeshFormer 0.966/0.052, with PSNR 21.692 vs. 20.500.
- On OmniObject3D, ARM achieves F-score 0.936, CD 0.067, and PSNR 20.874, outperforming MeshFormer (0.927/0.072/19.402).
- For relighted appearance, ARM reaches PSNR-A 21.750 vs. SF3D 18.592 on the synthetic relighting dataset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ARM rather than MeshFormer | single-image 3D reconstruction · GSO | PSNR: 20.500 -> 21.692 | Helped |
| Use ARM rather than MeshFormer | single-image 3D reconstruction · OmniObject3D | PSNR: 19.402 -> 20.874 | Helped |
| Use ARM rather than SF3D | relightable appearance reconstruction · Relighting dataset | PSNR-A: 18.592 -> 21.750 | Helped |
| Add back-projected image measurements (ablation) | ARM InstantAlbedo · Relighting dataset | PSNR-A: 24.780 -> 25.074 | Helped |
| Add material prior features (ablation) | ARM InstantAlbedo · Relighting dataset | PSNR-D: 22.687 -> 24.116 | Helped |
| Use FFC-Net with a global receptive field instead of a local-receptive-field U-Net (ablation) | ARM InstantAlbedo · Relighting dataset | LPIPS-D: 0.123 -> 0.098 | Helped |
| Decouple geometry and appearance prediction rather than predict all targets in one LRM (ablation) | ARM · 3D reconstruction | texture quality | Helped |
Surprising findings
- Predicting all targets (density, color, material) in a single LRM causes pronounced blurring, motivating decoupling.
- Removing the material prior hurts diffuse albedo decomposition, and jointly predicting roughness/metalness with albedo led to inaccuracies, so albedo was decomposed separately.
Limitations
- Inconsistencies in multi-view images from upstream diffusion models can introduce texture artifacts.
- Only six input views are used, leaving unobserved surface regions that require inpainting.
Experimental setup
Datasets: Objaverse GSO OmniObject3D custom synthetic PBR relighting dataset (100 objects)
Benchmarks: GSO OmniObject3D custom synthetic PBR relighting dataset
Metrics: F-Score Chamfer Distance PSNR SSIM LPIPS
Baselines: LGM CRM InstantMesh MeshFormer SF3D
Builds on: Large Reconstruction Model (LRM) InstantMesh Zero-1-to-3 / Zero123++ DINO FFC (fast Fourier convolution) DiffMC
Closest papers by meaning
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.87)
- ARMesh: Autoregressive Mesh Generation via Next-Level-of-Detail Prediction (NeurIPS 2025 · similarity 0.86)
- GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement (ICLR 2025 · similarity 0.86)
- SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement (CVPR 2025 · similarity 0.86)
- MeshGen: Generating PBR Textured Mesh with Render-Enhanced Auto-Encoder and Generative Data Augmentation (CVPR 2025 · similarity 0.86)
- IPDreamer: Appearance-Controllable 3D Object Generation with Complex Image Prompts (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.