Research map / 3D vision / Depth and stereo
Alligat0R: Pre-Training through Covisibility Segmentation for Relative Camera Pose Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces CroCo's cross-view completion pre-training with per-pixel covisibility segmentation (covisible/occluded/outside FOV) on a new 5M-pair Cub3 dataset, then fine-tunes for metric relative pose regression.
Problem
Cross-view completion pre-training (CroCo) is ill-posed in non-covisible regions, so it works only on pairs with substantial overlap, limiting its effectiveness for binocular tasks such as relative pose regression.
Approach
Pre-train a symmetric encoder-decoder without masking to classify each pixel as covisible, occluded, or outside FOV relative to the other view, using dense covisibility labels derived from depth and poses; the segmentation head is kept during pose fine-tuning.
Key results
- On RUBIK, Alligat0R pre-trained and fine-tuned on Cub3-all with frozen backbone reaches 55.3% at 5°/2m and 82.3% at 10°/5m, versus 8.9% and 25.2% for CroCo on the same data.
- On ScanNet1500 with unfrozen backbone, Alligat0R (Cub3-all) reaches 85.5 / 92.5 / 95.1 at 10°/0.25m, 0.5m, 1m, versus 75.7 / 87.4 / 91.5 for CroCo (Cub3-50).
- Overall RUBIK success rate of 60.3% ranks first among compared methods including MASt3R (53.6%) and RoMa (47.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace CroCo cross-view completion with covisibility segmentation pre-training on Cub3-all | Alligat0R versus CroCo, frozen backbone · RUBIK | success rate at 5° / 2m: 8.9 -> 55.3 | Helped |
| Replace CroCo cross-view completion with covisibility segmentation pre-training on Cub3-all | Alligat0R versus CroCo, frozen backbone · RUBIK | success rate at 10° / 5m: 25.2 -> 82.3 | Helped |
| Replace CroCo cross-view completion with covisibility segmentation pre-training on Cub3-all | Alligat0R versus CroCo, unfrozen backbone · ScanNet1500 | success rate at 10° / 0.25m: 64.1 -> 85.5 | Helped |
| Train Alligat0R on Cub3-all rather than Cub3-50 | Alligat0R, frozen backbone · RUBIK | success rate at 5° / 2m: 32.4 -> 55.3 | Helped |
| Train CroCo on Cub3-all rather than Cub3-50 | CroCo, frozen backbone · RUBIK | success rate at 5° / 2m: 21.8 -> 8.9 | Hurt |
| Inject 20% random label noise during pre-training (ablation) | Alligat0R · ScanNet1500 | success rate at 10° / 0.25m: 85.5 -> 87.7 (+2.2%) | Helped |
| Inject 20% random label noise during pre-training (ablation) | Alligat0R · ScanNet1500 | success rate at 10° / 0.5m: 92.5 -> 94.9 (+2.4%) | Helped |
| Inject 20% random label noise during pre-training (ablation) | Alligat0R · ScanNet1500 | success rate at 10° / 1m: 95.1 -> 95.9 (+0.8%) | Helped |
| Merge occluded and outside-FOV classes into one class (ablation) | Alligat0R · RUBIK | success rate at 5° / 0.5m: 22.5 vs 24.6 | Hurt |
| Merge occluded and outside-FOV classes into one class (ablation) | Alligat0R · RUBIK | success rate at 10° / 5m: 80.0 vs 81.9 | Hurt |
| Merge covisible and occluded classes into one class (ablation) | Alligat0R · RUBIK | success rate at 5° / 2m: 59.6 vs 60.3 | Hurt |
| Merge covisible and occluded classes into one class (ablation) | Alligat0R · RUBIK | success rate at 5° / 0.5m: 24.6 vs 24.6 | No clear effect |
Surprising findings
- Adding 20% random label noise during ScanNet pre-training improved Alligat0R performance across thresholds, which the authors attribute to a label-smoothing-like regularization effect.
- CroCo performed similarly or better when trained on easier pairs (Cub3-50) than on the full distribution (Cub3-all), whereas Alligat0R improved with the harder full distribution.
Limitations
- Evaluation limited to urban driving (nuScenes) and indoor (ScanNet) scenes.
- Pre-training requires ground-truth or pseudo ground-truth depth and poses to build covisibility labels, which is computationally intensive for nuScenes.
- Effectiveness on other binocular tasks such as stereo matching or optical flow is not explored.
Experimental setup
Datasets: Cub3 nuScenes ScanNet RUBIK ScanNet1500 ETH3D
Benchmarks: RUBIK ScanNet1500 ETH3D
Metrics: success rate at 5°/2m success rate at 10°/5m AUC@5 AUC@10 AUC@20 AEPE
Baselines: CroCo CroCo v2 From scratch LoFTR ELoFTR ASpanFormer RoMa DUSt3R MASt3R
Builds on: CroCo DUSt3R Reloc3R RUBIK
Closest papers by meaning
- Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors (CVPR 2025 · similarity 0.85)
- Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization (CVPR 2025 · similarity 0.85)
- 3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation (ICLR 2024 · similarity 0.84)
- NViST: In the Wild New View Synthesis from a Single Image with Transformers (CVPR 2024 · similarity 0.84)
- Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos (CVPR 2024 · similarity 0.84)
- No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.