Research map / 3D vision / Depth and stereo
LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRA3D, a self-calibration pipeline that specializes DUSt3R (and MASt3R) to a target scene using robust multi-view global optimization with confidence recalibration, pseudo-labeling, and rank-16 LoRA fine-tuning.
Problem
Pre-trained 3D geometric foundation models such as DUSt3R degrade under challenging conditions (limited view overlap, low light) and their predicted confidence is often miscalibrated, while annotated in-the-wild 3D data is scarce.
Approach
Treat the confidence maps as optimizable weights in a multi-view 3D-3D global alignment, with a regularizer tying weights to predicted confidence (Geman-McClure-like); closed-form IRLS-style update yields calibrated confidence that selects high-quality pseudo labels for LoRA fine-tuning on the target scene's own predictions, with no labels or external priors.
Key results
- On Replica, self-calibration reduces DUSt3R pairwise point prediction error by up to 38% and yields reconstructions up to 61% more accurate and 41% more complete than DUSt3R-Pretrain.
- On Waymo, camera trajectory error is reduced by up to 88% and focal error by up to 79% relative to DUSt3R-Pretrain; improved on 116 of 150 scenes.
- Self-calibration takes under 3.5 minutes on a single 3090 GPU with rank-16 LoRA (peak memory under 20GB; adapter under 18MB).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| self-calibrate DUSt3R using calibrated-confidence pseudo-labeling and LoRA | DUSt3R-Pretrain · Replica pairwise reconstruction | average point prediction error: 14.29 -> 8.84 | Helped |
| self-calibrate DUSt3R using calibrated-confidence pseudo-labeling and LoRA | DUSt3R-Pretrain · Waymo camera estimation | ATE: 0.79 -> 0.37 | Helped |
| self-calibrate DUSt3R and use it to initialize InstantSplat | InstantSplat with DUSt3R-Pretrain initialization · Waymo segment-10084 novel view rendering | PSNR: 21.45 -> 22.42 | Helped |
| self-calibrate MASt3R with pseudo-labeled calibration views | MASt3R-Pretrain · Replica office0 multi-view reconstruction | accuracy: 4.69 -> 4.61 | Helped |
| use un-calibrated prediction confidence for pseudo-label selection instead of calibrated confidence (ablation) | DUSt3R self-calibration · Replica office0 and Waymo segment-10084 | model performance | Hurt |
| vary the random seed for calibration-image sampling (ablation) | DUSt3R self-calibration · calibration-image sampling | model performance | No clear effect |
| use fewer than 10 calibration images instead of around 10 (ablation) | DUSt3R self-calibration · calibration-image count | model performance | Hurt |
| increase calibration images beyond around 10 (ablation) | DUSt3R self-calibration · calibration-image count | model performance | No clear effect |
| reduce the calibration split to the first 10 to 50 images (ablation) | DUSt3R self-calibration · Replica and Waymo calibration splits | model performance | Hurt |
| exclude office0 pseudo-labeled data from concurrent fine-tuning (ablation) | DUSt3R concurrent multi-scene self-calibration · Replica office0 | average point prediction error: 8.83 -> 11.64 | Hurt |
| exclude office0 pseudo-labeled data from concurrent fine-tuning (ablation) | DUSt3R concurrent multi-scene self-calibration · Replica office0 | accuracy: 4.43 -> 6.11 | Hurt |
| exclude office0 pseudo-labeled data from concurrent fine-tuning (ablation) | DUSt3R concurrent multi-scene self-calibration · Replica office0 | completeness: 6.08 -> 8.74 | Hurt |
Surprising findings
- Pseudo-labeling with uncalibrated (raw) DUSt3R confidence consistently hurts performance, regardless of cutoff.
- Using DUSt3R predictions to initialize COLMAP bundle adjustment significantly underperforms the DUSt3R-Pretrain baseline.
Limitations
- Gains for MASt3R self-calibration are mostly marginal, attributed to unfiltered noise in pseudo labels.
- Self-calibrated model still falls short of competing methods (e.g., MASt3R/COLMAP) in some cases.
- Degenerate static-vehicle sequences yield no improvement; novel view evaluation limited to mostly static segments.
Experimental setup
Datasets: Replica Waymo Open Dataset TUM RGBD
Benchmarks: Replica TUM RGBD Waymo Open Dataset
Metrics: average point prediction error accuracy completeness ATE AFE PSNR SSIM LPIPS
Baselines: DUSt3R-Pretrain DUSt3R-GT-FT COLMAP FlowMap MASt3R RelPose++ PoseDiffusion RayDiffusion
Builds on: DUSt3R LoRA MASt3R InstantSplat Geman-McClure robust M-estimator
Closest papers by meaning
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting for Novel View Synthesis (ICML 2025 · similarity 0.88)
- DUSt3R: Geometric 3D Vision Made Easy (CVPR 2024 · similarity 0.87)
- Fin3R: Fine-tuning Feed-forward 3D Reconstruction Models via Monocular Knowledge Distillation (NeurIPS 2025 · similarity 0.87)
- LRM-Zero: Training Large Reconstruction Models with Synthesized Data (NeurIPS 2024 · similarity 0.87)
- MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds (CVPR 2025 · similarity 0.87)
- Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.