Research map / 3D vision / Depth and stereo
Extreme Rotation Estimation in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Transformer-based relative rotation estimator trained with progressive learning (FoV and diffusion appearance augmentations, then real Internet pairs) and a new ExtremeLandmarkPairs benchmark for non-overlapping in-the-wild image pairs.
Problem
Estimating relative 3D rotation between extreme-view (limited or non-overlapping) Internet photos that vary in appearance and camera intrinsics, where prior methods only handle panorama-cropped images with constant intrinsics and illumination.
Approach
A Transformer decoder over frozen LoFTR features plus auxiliary keypoint/match and segmentation channels predicts discretized Euler angles, trained via a progressive scheme that moves from panorama crops to FoV-varied, diffusion-augmented, and finally real Internet pairs.
Key results
- On wELP large-overlap pairs, the model reaches 97.5 RRA15 and 2.41° MGE, comparable to Dust3R (98.4 RRA15, 1.01° MGE) and above SIFT/LoFTR-based methods in success rate.
- On wELP non-overlapping pairs, the model achieves 36.1 RRA15 and 26.97° MGE, versus 15.4 RRA15 for Dust3R and 7.5 RRA15 for CascadedAtt.
- Progressive training reduces wELP small-overlap median error from 55.28° (prior DenseCorrVol-style init) to 4.47°, with the ELP stage giving the largest gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FoV augmentation (+∆FoV) after panorama initialization (ablation) | Ours · wELP Small | MGE: 55.28 -> 12.91 | Helped |
| Add image-level appearance augmentation (+∆Im) after FoV augmentation (ablation) | Ours · wELP Small | MGE: 12.91 -> 11.46 | Helped |
| Train on ExtremeLandmarkPairs after appearance augmentation (ablation) | Ours · wELP Small | MGE: 11.46 -> 4.47 | Helped |
| Train on ExtremeLandmarkPairs after appearance augmentation (ablation) | Ours · wELP None | MGE: 68.31 -> 26.97 | Helped |
| Add keypoint and match masks (KP) (ablation) | Ours · wELP None | MGE: 41.39 -> 26.97 | Helped |
| Add segmentation maps (SM) (ablation) | Ours · wELP None | MGE: 43.07 -> 26.97 | Helped |
| Use the proposed method rather than Dust3R | Rotation estimation · sELP None | MGE: 114.33 -> 13.62 | Helped |
| Use the proposed method rather than Dust3R | Rotation estimation · sELP Large | MGE: 0.77 -> 2.45 | Hurt |
| Use the proposed method rather than Dust3R | Rotation estimation · wELP None | MGE: 81.21 -> 26.97 | Helped |
Surprising findings
- Dust3R, despite being trained on MegaDepth which overlaps the wELP test source, performs very poorly on non-overlapping pairs (RRA15 15.4), though it is strongest on overlapping pairs.
- Top-5 peak predictions substantially reduce non-overlap median error (27.0° to 12.8°), suggesting the model learns the answer but the largest peak is not always correct.
Limitations
- Non-overlapping pair performance remains high-error (median ~27° on wELP).
- Relies on pretrained LoFTR features and needs enough overlap for feature matching in some cases; correspondence baselines fail to produce poses for many non-overlapping pairs.
- Baseline CascadedAtt was reproduced because no pretrained model was released.
Experimental setup
Datasets: StreetLearn MegaScenes MegaDepth Cambridge Landmarks
Benchmarks: ExtremeLandmarkPairs sELP ExtremeLandmarkPairs wELP
Metrics: median geodesic error RRA15 RRA30 top-5 prediction error
Baselines: SIFT LoFTR DenseCorrVol CascadedAtt 8PointViT Dust3R
Builds on: LoFTR Cai et al. DenseCorrVol CascadedAtt InstructPix2Pix MegaScenes
Closest papers by meaning
- Estimating Extreme 3D Image Rotations using Cascaded Attention (CVPR 2024 · similarity 0.87)
- Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models (ICML 2025 · similarity 0.84)
- AerialMegaDepth: Learning Aerial-Ground Reconstruction and View Synthesis (CVPR 2025 · similarity 0.84)
- WildCAT3D: Appearance-Aware Multi-View Diffusion in the Wild (NeurIPS 2025 · similarity 0.83)
- UniK3D: Universal Camera Monocular 3D Estimation (CVPR 2025 · similarity 0.83)
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.