Research map / 3D vision / Depth and stereo
DiffCalib: Reformulating Monocular Camera Calibration as Diffusion-Based Dense Incident Map Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates monocular camera intrinsic estimation as diffusion-based generation of a dense incident map (jointly with depth) fine-tuned from Stable Diffusion v2.1, with RANSAC recovering the 4-DoF intrinsics.
Problem
Monocular camera calibration (estimating the 4-DoF intrinsics from a single image) generalizes poorly across scenes because existing methods rely on geometric or object assumptions and training data is limited.
Approach
Treat the per-pixel incident map as an image-like target and fine-tune a Stable Diffusion U-Net (frozen VAE) to generate it jointly with a depth map, using the joined latents as a 3D representation; intrinsics are then recovered by non-learning RANSAC, with ensembling over noise seeds for confidence.
Key results
- On zero-shot Waymo, DiffCalib reduces focal error e_f to 0.145 vs 0.210 for WildCamera (about 31% reduction); with the assumption-based variant e_f reaches 0.042 on ScanNet vs 0.109 for WildCamera+Asm.
- On ScanNet 3D reconstruction, F-score improves to 0.508 vs 0.377 for LeReS (about 30% higher), while Chamfer L1 is worse (0.150 vs 0.126).
- Joint depth and incident map training improves depth AbsRel on ETH3D from 0.101 to 0.086 compared with depth-only training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add depth-map joint training to incident-map estimation (ablation) | DiffCalib · Waymo | e_f: 0.188 -> 0.145 | Helped |
| add depth-map joint training to incident-map estimation (ablation) | DiffCalib · RGBD | e_f: 0.092 -> 0.084 | Helped |
| add depth-map joint training to incident-map estimation (ablation) | DiffCalib · ScanNet | e_f: 0.089 -> 0.055 | Helped |
| add depth-map joint training to incident-map estimation (ablation) | DiffCalib · Scenes11 | e_f: 0.108 -> 0.176 | Hurt |
| add depth-map joint training to incident-map estimation (ablation) | DiffCalib · RGBD | e_b: 0.018 -> 0.040 | Hurt |
| use U-Net pre-training (ablation) | DiffCalib · Waymo | e_f: 0.263 -> 0.188 | Helped |
| use U-Net pre-training (ablation) | DiffCalib · Scenes11 | e_f: 0.214 -> 0.108 | Helped |
| ensemble predictions from 10 noise samples (ablation) | DiffCalib · Waymo | e_f: 0.160 -> 0.145 | Helped |
| add incident-map supervision to depth training (ablation) | DiffCalib · ETH3D | AbsRel: 0.101 -> 0.086 | Helped |
| add incident-map supervision to depth training (ablation) | DiffCalib · NYU indoor | AbsRel: 0.086 -> 0.082 | Helped |
| jointly train surface-normal maps with incident maps (ablation) | DiffCalib · ScanNet | e_f: 0.071 -> 0.041 | Helped |
| jointly train surface-normal maps with incident maps (ablation) | DiffCalib · MVS | e_f: 0.112 -> 0.132 | Hurt |
Surprising findings
- Training with normal maps instead of depth also improves incident map prediction on several in-the-wild datasets, suggesting any dense geometric signal helps.
- Joint depth training hurts on RGBD and Scenes11 relative to incident-only training, which the authors attribute to unreliable manufacturer intrinsics and synthetic moving-object images.
Limitations
- Depth from joint training can fail in some conditions, degrading Chamfer L1 on ScanNet.
- Performance on object-level datasets (Objectron, MVImgNet) lags in focal length estimation.
- Depth reconstruction needs an external frozen shift model and assumes the optical center for some settings.
Experimental setup
Datasets: Hypersim NuScenes KITTI CitySpace NYUv2 SUN3D ARKitScenes Objectron MVImgNet Waymo RGBD ScanNet
Benchmarks: NuScenes KITTI CitySpace NYUv2 SUN3D ARKitScenes Objectron MVImgNet Waymo RGBD ScanNet MVS
Metrics: e_f (focal length error) e_b (optical center error) AbsRel delta1 Chamfer L1 F-score@5cm
Baselines: Perspective field WildCamera LeReS
Builds on: Stable Diffusion incidence field (Zhu et al.) WildCamera Marigold-style diffusion depth estimation RANSAC
Closest papers by meaning
- MonoDiff: Monocular 3D Object Detection and Pose Estimation with Diffusion Models (CVPR 2024 · similarity 0.88)
- Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (CVPR 2024 · similarity 0.87)
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (ICLR 2025 · similarity 0.86)
- MultiDiff: Consistent Novel View Synthesis from a Single Image (CVPR 2024 · similarity 0.86)
- MonoPlace3D: Learning 3D-Aware Object Placement for 3D Monocular Detection (CVPR 2025 · similarity 0.86)
- VGGT-SLAM: Dense RGB SLAM Optimized on the SL(4) Manifold (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.