Research map / 3D vision / Depth and stereo
AnyMap: Learning a General Camera Model for Structure-from-Motion with Unknown Distortion in Dynamic Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
AnyMap is a differentiable SfM framework that learns a general implicit camera model (an invertible neural network for unprojection/projection) plus low-dimensional basis-trajectory motion, handling lens distortion in dynamic videos.
Problem
Learning-based structure-from-motion typically assumes pinhole cameras and fails on videos of dynamic scenes captured with wide-angle or fisheye cameras with unknown lens distortion; motion and distortion estimates can compensate for each other.
Approach
Replace predefined camera models with an invertible neural network that learns the (un)projection functions end-to-end, and regularize per-pixel scene motion with a small set of learnable basis trajectories combined by MLP-predicted weights so motion does not absorb distortion errors.
Key results
- On rectification/calibration (Table 1), AnyMap reaches 28.15 PSNR and 0.021 LPIPS on MipNeRF-360 versus 22.85 PSNR for COLMAP MVS.
- On DAVIS dynamic scenes with distortion, AnyMap achieves 22.54 PSNR and ATE 0.106, outperforming MonST3R (ATE 0.214) and CasualSAM (ATE 0.228).
- Pose estimation ATE on Tanks & Temples is 0.0092 for AnyMap versus 0.0142 for COLMAP MVS, though AnyMap is 2.8x slower than COLMAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the learned camera model and differentiable SfM framework | AnyMap · MipNeRF-360, LLFF, Tanks & Temples, and Aria Everyday Activities | PSNR: 28.77 -> 32.08; 23.98 -> 23.32; 24.98 -> 26.33; 22.35 -> 25.03 | Helped |
| add the learned camera model and differentiable SfM framework | AnyMap · MipNeRF-360, LLFF, Tanks & Temples, and Aria Everyday Activities | ATE: 0.0023 -> 0.0022; 0.0042 -> 0.0034; 0.0142 -> 0.0092; 0.0120 -> 0.0101 | Helped |
| add the unprojection loss Lu (ablation) | AnyMap · MipNeRF-360, LLFF, Tanks & Temples, and Aria Everyday Activities | LPIPS: 0.035 -> 0.021; 0.080 -> 0.069; 0.059 -> 0.056; 0.087 -> 0.076 | Helped |
| allow dynamic motion in place of the static-scene constraint (ablation) | AnyMap · MipNeRF-360, LLFF, Tanks & Temples, and Aria Everyday Activities | ATE: 0.0024 -> 0.0022; 0.0040 -> 0.0034; 0.0110 -> 0.0092; 0.0109 -> 0.0101 | Mixed |
| replace basis-trajectory motion with dense CVD motion prediction (ablation) | AnyMap · DAVIS | Repx: 3.20 -> 8.21 | Hurt |
| remove scene-motion estimation (ablation) | AnyMap · DAVIS | Repx: 3.20 -> 11.03 | Hurt |
| replace the learned camera network with parameterized camera-model functions (ablation) | AnyMap · MipNeRF-360 | LPIPS: 0.021 -> 0.078 | Hurt |
| replace pinhole unprojection with the AnyMap camera network | CasualSAM · DAVIS | Repx: 9.23 -> 8.47 | Helped |
| add monocular depth loss Lm (ablation) | AnyMap · dynamic scenes | scale drift | Helped |
Surprising findings
- Directly optimizing parameterized Fisheye624/KB6 intrinsics performs much worse than the learned camera model, due to optimization instability from camera model ambiguities.
- Unregularized dense motion tends to compensate for unhandled distortion; CasualSAM predicts compensating motion at image borders.
Limitations
- Slower and more memory-intensive than scene coordinate regressors such as ACE0.
- Requires optical flow and point tracks, so operates only on video sequences.
- Dominant motion can cause static backgrounds to be treated as moving, mitigated only by motion masks.
Experimental setup
Datasets: Aria Everyday Activities MipNeRF-360 LLFF Tanks & Temples DAVIS
Benchmarks: Aria Everyday Activities MipNeRF-360 LLFF Tanks & Temples DAVIS
Metrics: PSNR SSIM LPIPS relative focal length error reprojection error ATE Abs Rel delta<1.25 runtime
Baselines: COLMAP GLOMAP COLMAP MVS GeoCalib Porfiri et al. Auto-DC PCN DROID-SLAM MASt3R MonST3R CasualSAM
Builds on: FlowMap FlowCam CasualSAM Consistent Video Depth ResFlow
Closest papers by meaning
- AnyCam: Learning to Recover Camera Poses and Intrinsics from Casual Videos (CVPR 2025 · similarity 0.90)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.86)
- VGGSfM: Visual Geometry Grounded Deep Structure From Motion (CVPR 2024 · similarity 0.86)
- No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images (ICLR 2025 · similarity 0.85)
- Light3R-SfM: Towards Feed-forward Structure-from-Motion (CVPR 2025 · similarity 0.85)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.