Research map / 3D vision / Human pose estimation
AnyCam: Learning to Recover Camera Poses and Intrinsics from Casual Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AnyCam, a transformer that feed-forward predicts camera poses, intrinsics (via focal-length hypotheses), and uncertainty maps from casual videos, trained without labels using depth/flow priors, with lightweight BA refinement.
Problem
Estimating camera poses and intrinsics from casual, dynamic videos is hard for SfM/SLAM, and learned alternatives either need intrinsics, labeled supervision, or slow test-time optimization.
Approach
A feed-forward transformer predicts relative poses per frame pair with uncertainty maps and selects focal length among multiple candidate prediction heads; training uses an uncertainty-weighted flow-induced reprojection loss, forward-backward pose consistency, and a KL loss over focal candidates, so no pose labels are needed; a lightweight uncertainty-weighted bundle adjustment refines the traject
Key results
- On Sintel, AnyCam with refinement achieves ATE 0.078 vs. 0.141 for CasualSAM and 0.108 for MonST3R, with RPErot 0.427 vs. 0.615 for CasualSAM.
- Intrinsics estimation on Sintel: mean absolute focal error 252.2 px vs. 434.0 px for Dust3r and 447.4 px for UniDepth.
- Refinement takes about 90 s for a 50-frame video, and the full pipeline is under 2 minutes, versus over 1 hour for CasualSAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AnyCam's pose and intrinsic prediction approach | Camera estimation · Sintel | ATE: 0.089 -> 0.078 | Helped |
| Add AnyCam's pose and intrinsic prediction approach | Camera estimation · TUM-RGBD (dynamics) | RPEtrans: 0.008 -> 0.005 | Helped |
| Use video-based focal length recovery | AnyCam · Sintel | AFE (px): 447.4 -> 252.2 | Helped |
| Train and infer using sequences of 2 frames instead of the full sequence setup (ablation) | AnyCam · Sintel | ATE: 0.078 -> 0.134 | Hurt |
| Train with sequence length 2 and infer with sequence length ≥8 (ablation) | AnyCam · Sintel | ATE: 0.078 -> 0.179 | Hurt |
| Train with sequence length 8 but infer with sequence length 2 (ablation) | AnyCam · Sintel | ATE: 0.099 -> 0.182 | Hurt |
| Add simple bundle-adjustment refinement without uncertainty weighting (ablation) | AnyCam · Sintel | ATE: 0.099 -> 0.136 | Hurt |
| Add bundle-adjustment refinement weighted by predicted uncertainty (ablation) | AnyCam · Sintel | ATE: 0.099 -> 0.078 | Helped |
| Add bundle-adjustment refinement weighted by predicted uncertainty (ablation) | AnyCam · Sintel | RPEtrans: 0.045 -> 0.031 | Helped |
Surprising findings
- Naive bundle adjustment does not improve the predicted trajectory; only weighting pixel tracks by the predicted uncertainty makes refinement help.
- Directly regressing focal length made training unstable, so the model instead uses discrete focal-length candidate heads.
Limitations
- ATE drifts without test-time refinement because relative poses are chained.
- Focal length selection via the flow loss can be noisy and sometimes picks a wrong candidate.
Experimental setup
Datasets: YouTube-VOS RealEstate10K WalkingTours OpenDV EpicKitchens Sintel TUM-RGBD Davis Aria Everyday Activities Waymo
Benchmarks: Sintel TUM-RGBD (dynamics) Davis Waymo Aria Everyday Activities
Metrics: ATE RPE translation RPE rotation absolute focal error (AFE) relative focal error (RFE)
Baselines: DROID-SLAM DPVO LeapVO ParticleSfM MonST3R Robust-CVD CasualSAM UniDepth Dust3r
Builds on: DINOv2 UniDepth UniMatch DROID-SLAM FlowMap
Closest papers by meaning
- AnyMap: Learning a General Camera Model for Structure-from-Motion with Unknown Distortion in Dynamic Scenes (CVPR 2025 · similarity 0.90)
- InFlux: A Benchmark for Self-Calibration of Dynamic Intrinsics of Video Cameras (NeurIPS 2025 · similarity 0.86)
- MegaSaM: Accurate, Fast and Robust Structure and Motion from Casual Dynamic Videos (CVPR 2025 · similarity 0.86)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.85)
- Free-Moving Object Reconstruction and Pose Estimation with Virtual Camera (AAAI 2025 · similarity 0.85)
- Dynamic Camera Poses and Where to Find Them (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.