Research map / 3D vision / 3D generation
Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STRING, a general family of separable, exactly translation-invariant position encodings built from commuting learnable skew-symmetric generators, generalizing RoPE to 2D/3D token coordinates for vision and robotics transformers.
Problem
Rotary position encodings (RoPE) are limited in higher-dimensional position coordinates: multi-axis RoPE is restricted to axis-aligned rotations, and more general dense SO(d) alternatives break translational invariance for d_c>1 and have large memory footprints, which hampers robotics and 2D/3D visi
Approach
Define position encodings as R(r)=exp(sum_k L_k r_k) with learnable commuting skew-symmetric generators; prove this is the most general translation-invariant matrix-multiplication PE and that RoPE is a special case. Make it efficient via a change of basis P (Cayley-STRING) or circulant generators computed with FFT (Circulant-STRING).
Key results
- Cayley-STRING achieves best mean on classification (69.12 vs RoPE 68.58 mean, ViT 68.42) and best average on WebLI retrieval (70.48 vs RoPE 69.86).
- On 2D open-vocabulary detection (OWL-ViT), Cayley-STRING reaches mean AP 28.24 vs RoPE 28.18 and baseline 27.21.
- On ALOHA simulation MultiTask success, Cayley-STRING achieves 0.4583 vs RoPE 0.4167 and ViT 0.3667.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ViT position encoding with Circulant-STRING | ViT-B/16 · ImageNet2012 | top-1 test accuracy (%): 80.86 -> 81.22 | Helped |
| Replace ViT position encoding with Cayley-STRING | ViT-B/16 · Places365 | top-1 test accuracy (%): 56.97 -> 57.16 | Helped |
| Replace RoPE with Cayley-STRING | ViT-B/16 · WebLI-3D | mean retrieval recall (%): 69.86 -> 70.48 | Helped |
| Replace RoPE with Cayley-STRING | OWL-ViT B/32 · COCO | average precision (%): 33.66 -> 33.47 | Hurt |
| Replace RoPE with Cayley-STRING | OWL-ViT B/32 · LVIS | average precision (%): 22.71 -> 23.01 | Helped |
| Replace RoPE with Circulant-STRING | ViT · 3D object detection | average 3D IoU (%): 58.091 -> 58.949 | Helped |
| Replace RoPE with Cayley-STRING | ViTD · 3D object detection | average 3D IoU (%): 71.206 -> 72.665 | Helped |
| Replace RoPE with Cayley-STRING | SigLIP B/16 256 ViT · ALOHA simulation MultiTask | mean success rate: 0.4167 -> 0.4583 | Helped |
| Add depth through normal maps | KUKA 3D robot policy · KUKA real-world regular evaluations | success rate | Hurt |
| Add depth by lifting patches to 3D and applying STRING | KUKA 2D base policy · KUKA real-world regular evaluations | success rate | Helped |
| Combine STRING with noisy normal maps | KUKA 3D robot policy · KUKA real-world regular evaluations | success rate | Helped |
| Use 3D STRING policy instead of the 2D policy | KUKA robot policy · OOD table-height variation | success rate: ≈ 10% -> ≈ 50% | Helped |
Surprising findings
- Cayley-STRING outperforms RoPE comprehensively even though STRING and RoPE are related by a fixed change of basis, which the authors attribute to learning the basis explicitly via a shared orthogonal matrix P.
- Adding normal-map depth inputs to the KUKA policy hurt on-robot performance, while STRING-based 3D position encoding improved it.
Limitations
- Gains over RoPE in several experiments are small (often under 1 percentage point on classification and retrieval).
- 3D detection evaluated on only 80 held-out images and results depend on max over 3 seeds due to training volatility.
- Real-robot and ALOHA results rely on proprietary or unreleased setups and limited trials (e.g., 50 A/B trials, 10 sim trials per task).
Experimental setup
Datasets: ImageNet2012 Places365 WebLI WebLI-3D ABO YCB Google Scanned Objects COCO LVIS ALOHA Unleashed simulation data
Benchmarks: ImageNet2012 Places365 WebLI retrieval COCO LVIS simulated 3D bounding box detection ALOHA simulation (12 tasks + MultiTask) KUKA real-robot out-of-distribution tests
Metrics: top-1 accuracy recall@k average precision 3D IOU success rate
Baselines: ViT (absolute position encoding) RoPE RoPE-Mixed OWL-ViT baseline SigLIP B/16 ViT
Builds on: RoPE RoPE-Mixed Cayley transform OWL-ViT SigLIP
Closest papers by meaning
- What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.86)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.86)
- A Circular Argument: Does RoPE need to be Equivariant for Vision? (NeurIPS 2025 · similarity 0.86)
- Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Vision-Language Models (ICML 2026 · similarity 0.86)
- LieRE: Lie Rotational Positional Encodings (ICML 2025 · similarity 0.85)
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.