Research map / Architectures and efficiency / RNNs and normalization
A Circular Argument: Does RoPE need to be Equivariant for Vision?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper proves RoPE and Mixed RoPE are general solutions for equivariant positional embeddings, proposes non-equivariant Spherical RoPE and Uniform-Frequency RoPE, and finds on ViT ImageNet/CIFAR100 that equivariance contributes little to RoPE's gains.
Problem
Whether RoPE's success in vision stems from positional equivariance (relative encoding), and whether multi-dimensional RoPE extensions must preserve equivariance.
Approach
Shows 1D LieRE equals RoPE with learned frequencies, that Mixed RoPE is the most general equivariant M-D rotary embedding (commutative generators), and proposes Spherical RoPE (non-commutative Euler-angle rotations on triplets) to test whether breaking equivariance hurts performance.
Key results
- Spherical RoPE performs comparably to Mixed RoPE and outperforms Axial RoPE on ImageNet and CIFAR100 with ViT-S.
- All RoPE variants outperform learned absolute positional encodings, including Uniform RoPE with a single shared frequency.
- At higher image resolutions, Spherical RoPE is the most effective method in the reported resolution-generalization experiment on ImageNet-1k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use non-equivariant Spherical RoPE instead of Axial RoPE | ViT-S · CIFAR100 and ImageNet-1K at training resolution | accuracy | Helped |
| Use non-equivariant Spherical RoPE instead of equivariant Mixed RoPE | ViT-S · CIFAR100 and ImageNet-1K at training resolution | accuracy | No clear effect |
| Replace learned absolute positional embeddings with RoPE | ViT-S · CIFAR100 and ImageNet-1K at training resolution | accuracy | Helped |
| Use Uniform RoPE instead of learned absolute positional embeddings | ViT-S · CIFAR100 and ImageNet-1K at training resolution | accuracy | Helped |
| Use diverse rotation frequencies instead of one shared frequency (ablation) | ViT-S with RoPE · CIFAR100 and ImageNet-1K at training resolution | accuracy | Helped |
| Use Spherical RoPE instead of other positional encodings for higher-resolution evaluation | ViT-S · ImageNet-1K at higher resolutions | accuracy | Helped |
| Use Spherical RoPE instead of Axial RoPE for resolution extrapolation | ViT-S · ImageNet-1K at higher resolutions | accuracy | Helped |
| Use Uniform RoPE instead of RoPE with diverse frequencies (ablation) | ViT-S · ImageNet-1K at higher resolutions | accuracy | Hurt |
| Use Learned Spherical RoPE instead of the other positional encoding methods | ViT-S · Low-data regime | accuracy | Helped |
Surprising findings
- Non-equivariant Spherical RoPE matches equivariant Mixed RoPE, suggesting strict equivariance is not a major driver of RoPE's vision gains.
- Uniform-frequency RoPE still beats learned absolute embeddings, suggesting an unidentified factor beyond the tested properties.
Limitations
- Experiments limited to small-scale vision (ViT-S, CIFAR100, ImageNet) with short context lengths, not language-scale settings.
- Proofs are informal and the paper notes LieRE-style results are not tuned for accuracy.
- Non-equivariance conclusions are drawn from vision only and may not transfer to physics or long-context tasks.
Experimental setup
Datasets: ImageNet-1K CIFAR100
Benchmarks: ImageNet-1K CIFAR100
Metrics: top-1 accuracy
Baselines: Learned APE Axial RoPE Mixed RoPE LieRE Uniform RoPE
Builds on: RoPE LieRE Mixed RoPE Axial RoPE DeiT-III
Closest papers by meaning
- Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Vision-Language Models (ICML 2026 · similarity 0.90)
- ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices (CVPR 2025 · similarity 0.88)
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.87)
- LieRE: Lie Rotational Positional Encodings (ICML 2025 · similarity 0.87)
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.87)
- Learning the RoPEs: Better 2D and 3D Position Encodings with STRING (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.