Research map / 3D vision / 3D generation
Fin3R: Fine-tuning Feed-forward 3D Reconstruction Models via Monocular Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fin3R fine-tunes feed-forward 3D reconstruction models by freezing the decoder and distilling MoGe monocular depth into the encoder via a re-normalization LoRA adapter, improving fine geometry while preserving multi-view performance.
Problem
Feed-forward 3D reconstruction models (DUSt3R, MASt3R, CUT3R, VGGT) produce coarse, over-smoothed geometry and degrade on long sequences, partly due to scarce high-fidelity depth/pose supervision and multi-view pointmap regression ambiguity.
Approach
Freeze the decoder and fine-tune only the image encoder with a lightweight LoRA adapter whose weights are re-normalized to preserve the original weight norm, distilling a monocular teacher's depth on unlabeled SA-1B images so that feature-norm drift does not break the frozen decoder's multi-view matching.
Key results
- VGGT+Ours improves ScanNet1500 relative pose AUC@5 from 28.40 to 35.21 and average monocular depth Rel from 5.77 to 4.29.
- DUSt3R+Ours improves average monocular depth Rel from 7.03 to 5.58 and RealEstate10k RRA@5 from 94.01 to 95.41.
- Ablation: re-normalization LoRA (35.21 AUC@5) outperforms plain LoRA (32.96) and full encoder fine-tuning (32.06) on ScanNet1500 with VGGT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Fin3R fine-tuning | DUSt3R · Monocular depth, Average | Rel: 7.03 -> 5.58 | Helped |
| Apply Fin3R fine-tuning | CUT3R · Video depth, Sintel | Rel: 0.428 -> 0.406 | Helped |
| Apply Fin3R fine-tuning | VGGT · ScanNet1500 | AUC@5: 28.40 -> 35.21 | Helped |
| Apply Fin3R fine-tuning | VGGT · 7-Scenes | Acc Mean: 0.017 -> 0.012 | Helped |
| Add depth-label supervision (ablation) | VGGT distillation · Distillation ablation | Rel: 5.68 -> 5.21 | Helped |
| Replace depth labels with monocular teacher supervision (ablation) | VGGT distillation · Distillation ablation | Rel: 5.21 -> 5.00 | Helped |
| Use SA-1B for monocular teacher distillation (ablation) | VGGT distillation · Distillation ablation | Rel: 5.00 -> 4.35 | Helped |
| Fine-tune the decoder with monocular data (ablation) | VGGT · ScanNet1500 | AUC@5: 28.40 -> 28.42 | No clear effect |
| Fully fine-tune the encoder (ablation) | VGGT · ScanNet1500 | AUC@5: 28.40 -> 32.06 | Helped |
| Fully fine-tune both encoder and decoder (ablation) | VGGT · ScanNet1500 | AUC@5: 28.40 -> 26.35 | Hurt |
| Use encoder LoRA without re-normalization (ablation) | VGGT · ScanNet1500 | AUC@5: 28.40 -> 32.96 | Helped |
| Add re-normalization to encoder LoRA (ablation) | VGGT · ScanNet1500 | AUC@5: 32.96 -> 35.21 | Helped |
Surprising findings
- Fine-tuning the decoder with monocular data harms multi-view consistency, even with the encoder also tuned (3) reduces AUC@5 to 26.35 versus 28.40 baseline.
- Fine-tuning VGGT's depth head with distillation also improves its pointmap head without direct supervision, suggesting a robustly trained encoder transfers across heads.
Limitations
- Boundary accuracy remains inferior to MoGe, the teacher model.
- Supports only 512/518 resolution; scaling to higher resolutions is unresolved.
- Re-normalization is not necessarily sufficient to address all feature shifts; CO3Dv2 pose performance for CUT3R degraded.
Experimental setup
Datasets: SA-1B Hypersim TartanAir MegaDepth CO3Dv2 ARKitScenes ScanNet++ ScanNet VirtualKITTI2 BlendedMVS StaticThings3D RealEstate10k
Benchmarks: NYUv2 KITTI ETH3D iBims-1 DDAD DIODE HAMMER ScanNet1500 RealEstate10k Sintel Bonn Tanks and Temples
Metrics: Rel delta_1 AUC@5 AUC@10 AUC@20 RRA@5 RTA@5 Acc Comp NC
Baselines: DUSt3R MASt3R CUT3R VGGT MoGe MoGe-2 Pi3 Reloc3r Efficient LoFTR ROMA NoPoSplat
Builds on: DUSt3R MASt3R CUT3R VGGT MoGe LoRA
Closest papers by meaning
- Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors (CVPR 2025 · similarity 0.88)
- SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images (CVPR 2025 · similarity 0.88)
- LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation models (ICLR 2025 · similarity 0.87)
- Test3R: Learning to Reconstruct 3D at Test Time (NeurIPS 2025 · similarity 0.87)
- DUSt3R: Geometric 3D Vision Made Easy (CVPR 2024 · similarity 0.87)
- SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.