Research map / Architectures and efficiency / Attention and transformers
ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ComRoPE, a generalization of rotary position embedding using trainable pairwise-commuting angle matrices, shown to improve ViT accuracy on ImageNet-1K and COCO detection over LieRE and APE.
Problem
Standard RoPE uses manually defined, non-trainable rotation matrices that limit expressiveness, and prior trainable variants (LieRE) do not consistently satisfy the relative-position (RoPE) equation, hurting robustness to resolution/position changes in vision transformers.
Approach
Formally defines the RoPE equation for angle-matrix-parameterized rotations and proves that pairwise commutativity of the angle matrices is necessary and sufficient; constructs trainable commuting matrices via axial-partition (ComRoPE-AP) or linearly dependent scaled base matrices (ComRoPE-LD).
Key results
- ComRoPE-LD outperforms LieRE by 1.6% at the 224 training resolution on ImageNet-1K.
- ComRoPE-LD outperforms LieRE by 2.9% at 512x512 resolution on ImageNet-1K.
- On COCO object detection with ViT-S, ComRoPE-LD reaches 44.7 AP vs 44.5 for LieRE and 44.0 for APE, with about half the extra parameters of LieRE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ComRoPE-LD instead of LieRE | ViT-B/16 · ImageNet-1K, 224×224 | accuracy: 1.6% | Helped |
| Use ComRoPE-LD instead of LieRE | ViT-B/16 · ImageNet-1K, 512×512 | accuracy: 2.9% | Helped |
| Use ComRoPE-LD instead of LieRE | ViT-S · MS COCO | AP: 44.5 -> 44.7 | Helped |
| Add coordinate offsets at inference (ablation) | LieRE · ImageNet-1K, varying resolutions | accuracy | Hurt |
| Add coordinate offsets at inference (ablation) | ComRoPE · ImageNet-1K, varying resolutions | accuracy | No clear effect |
| Increase block size (ablation) | LieRE, ComRoPE-AP, and ComRoPE-LD · ImageNet-1K, varying resolutions | accuracy | Helped |
| Increase position perturbation intensity from 0 to 1 (ablation) | APE · ImageNet-1K | accuracy: +19.5% | Helped |
| Increase position perturbation intensity from 0 to 1 (ablation) | ComRoPE-LD · ImageNet-1K | accuracy: +2.9% | Helped |
Surprising findings
- LieRE's rotation matrices do not always satisfy R(x_q)^T R(x_k) = R(x_q - x_k) because random matrices rarely commute, so LieRE does not consistently meet the RoPE equation.
Limitations
- Commutativity is only a sufficient construction; no general method gives necessary and sufficient trainable commuting matrices.
- Larger block sizes raise time complexity to O(Lndb^2), limiting block size to 8.
Experimental setup
Datasets: ImageNet-1K MS COCO
Benchmarks: ImageNet-1K MS COCO
Metrics: top-1 accuracy AP AP50 AP75 APs APm APl
Baselines: APE Vanilla RoPE LieRE
Builds on: RoFormer (RoPE) LieRE RoPE-Mixed
Closest papers by meaning
- LieRE: Lie Rotational Positional Encodings (ICML 2025 · similarity 0.89)
- A Circular Argument: Does RoPE need to be Equivariant for Vision? (NeurIPS 2025 · similarity 0.88)
- Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Vision-Language Models (ICML 2026 · similarity 0.88)
- What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.86)
- VideoRoPE: What Makes for Good Video Rotary Position Embedding? (ICML 2025 · similarity 0.86)
- DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.