Research map / 3D vision / Depth and stereo
Estimating Extreme 3D Image Rotations using Cascaded Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an attention-based network for extreme 3D relative rotation estimation that uses cross-decoding, a Transformer-Encoder cross-attention replacing the 4D correlation volume, and cascaded Transformer decoders with a learned quaternion query.
Problem
Estimating large, extreme relative 3D rotations between image pairs with limited or no field-of-view overlap, where feature-matching methods fail.
Approach
Replaces the 4D correlation volume with a Transformer-Encoder cross-attention computed via an attention mask that keeps only cross-image terms, refines image embeddings by cross-decoding, and alternately refines the cross-attention and a learned quaternion query through a cascade of two Transformer decoders, regressing the quaternion with an L2 loss.
Key results
- On StreetLearn large-overlap, the method reports 0.58 deg mean error and 99.31% under 10 deg, versus 1.19 deg for DenseCorrVol.
- On StreetLearn non-overlap pairs it reports 28.13 deg mean error versus 30.98 deg for DenseCorrVol.
- Cross-dataset generalization (Manhattan to London) gives 9.55 deg vs 11.23 deg for Cai et al. in the large-overlap class.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DenseCorrVol with cascaded attention | relative rotation estimation · InteriorNet None | mean geodesic error (°): 37.69 -> 35.13 | Helped |
| replace DenseCorrVol with cascaded attention | relative rotation estimation · SUN360 Small | mean geodesic error (°): 3.09 -> 2.109 | Helped |
| replace 8PointViT with cascaded attention | relative rotation estimation · InteriorNet Large | mean geodesic error (°): 0.48 -> 0.43 | Helped |
| use the proposed method instead of Cai et al. | Holicity cross-dataset evaluation, Manhattan to London · Large | average geodesic error (°): 11.23 -> 9.55 | Helped |
| use the proposed method instead of Cai et al. | Holicity cross-dataset evaluation, Manhattan to London · Small | average geodesic error (°): 20.87 -> 16.33 | Helped |
| use the proposed method instead of Cai et al. | Holicity cross-dataset evaluation, Manhattan to London · None | average geodesic error (°): 40.82 -> 38.48 | Helped |
| use 4 attention heads and 2 layers instead of 1 head and 1 layer (ablation) | Transformer-Encoder on StreetLearn · Large | mean geodesic error (°): 1.21 -> 0.58 | Helped |
| increase Transformer-Encoder from 2 to 4 layers (ablation) | Transformer-Encoder on StreetLearn · Large | mean geodesic error (°): 0.58 -> 0.65 | Hurt |
| use 3 residual blocks instead of 1 (ablation) | Residual-Unet backbone on StreetLearn · Large | mean geodesic error (°): 2.95 -> 1.21 | Helped |
| increase backbone from 3 to 4 residual blocks (ablation) | Residual-Unet backbone on StreetLearn · Large | mean geodesic error (°): 1.21 -> 1.45 | Hurt |
| use quaternion representation with L2 regression instead of Euler angles with Cross-Entropy (ablation) | rotation regression on StreetLearn · Large | mean geodesic error (°): 2.37 -> 1.21 | Helped |
| use Euler angles with L2 regression instead of quaternion L2 regression (ablation) | rotation regression on StreetLearn · Large | mean geodesic error (°): 1.21 -> 2.22 | Hurt |
Surprising findings
- Removing the cross-decoding (TD0) only modestly degrades results, while removing both decoders raises mean error from 0.58 to 3.35 deg on StreetLearn large overlap.
- Adding further cascade stages gave no performance gain, according to the authors.
Limitations
- Evaluated only on rotation estimation; applicability to other two-image tasks is future work.
- Relies on urban scene priors (vertical and horizontal lines) for non-overlapping pairs, per the authors' discussion.
- Images are 128x128 perspective crops and only yaw and pitch are estimated, assuming zero roll.
Experimental setup
Datasets: InteriorNet StreetLearn SUN360 Holicity
Benchmarks: InteriorNet InteriorNet-T SUN360 StreetLearn StreetLearn-T Holicity (cross-dataset)
Metrics: mean geodesic error median geodesic error percentage of pairs under 10 degrees
Baselines: SIFT SuperPoint Reg6D DenseCorrVol 8PointViT
Builds on: DenseCorrVol (Cai et al.) Transformer Residual U-Net 8PointViT
Closest papers by meaning
- Extreme Rotation Estimation in the Wild (CVPR 2025 · similarity 0.87)
- Selective Disk Bispectrum: A Complete and Rotation Invariant Image Descriptor (NeurIPS 2026 · similarity 0.84)
- Steerers: A Framework for Rotation Equivariant Keypoint Descriptors (CVPR 2024 · similarity 0.83)
- Cross-View Completion Models are Zero-shot Correspondence Estimators (CVPR 2025 · similarity 0.83)
- DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers (NeurIPS 2026 · similarity 0.82)
- 3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.