Research map / Architectures and efficiency / Attention and transformers
LieRE: Lie Rotational Positional Encodings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LieRE, a positional encoding that replaces RoPE's fixed block-diagonal 2D rotations with learned dense skew-symmetric matrices mapped via the matrix exponential to high-dimensional rotations, and shows gains on 2D/3D vision and a synthetic arrow-direction task.
Problem
RoPE is designed for one-dimensional sequences and uses fixed block-diagonal 2D rotations, which limits its ability to encode positional structure in high-dimensional data such as images and videos.
Approach
Learn a dense skew-symmetric basis per layer and head, compute position rotations R(p)=exp(sum_i p_i A_i) via the matrix exponential, and apply them to queries and keys; RoPE-Mixed is the special case of 2x2 blocks, and 1D LieRE is shown equivalent to RoPE up to a learnable basis.
Key results
- On CIFAR-100, LieRE_8 reaches 70.3% top-1 vs 68.8% for RoPE-Mixed and 63.9% for absolute encodings.
- On the synthetic arrow task at 276x276, LieRE_64 achieves 100% and LieRE_8 99.7%, while RoPE-Mixed drops to 88.6% and absolute encoding to 40.1%.
- On UCF101, LieRE_8 reaches 47.0% top-1 vs 46.3% for RoPE-Mixed and 40.9% for absolute encodings; LieRE adds 0.68% parameters to ViT-B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace absolute positional embeddings with LieRE₈ | ViT-B · CIFAR-100 | top-1 accuracy: 63.9 -> 70.3 | Helped |
| replace RoPE-Mixed with LieRE₈ | ViT-B · CIFAR-100 | top-1 accuracy: 68.8 -> 70.3 | Helped |
| replace RoPE-Mixed with LieRE₈ | ViT-B · ImageNet-1k | top-1 accuracy: 68.8 -> 69.6 | Helped |
| replace RoPE-Mixed with LieRE₈ | ViT-style 3D · UCF101 | top-1 accuracy: 46.3 -> 47.0 | Helped |
| increase LieRE basis block width from 8 to 64 (ablation) | LieRE · CIFAR-100 | top-1 accuracy: 70.3 -> 70.0 | Hurt |
| increase LieRE basis block width from 8 to 64 (ablation) | LieRE · ImageNet-1k | top-1 accuracy: 69.6 -> 69.3 | Hurt |
| increase LieRE basis block width from 8 to 64 (ablation) | LieRE · UCF101 | top-1 accuracy: 47.0 -> 44.7 | Hurt |
| increase LieRE basis block width from 8 to 64 (ablation) | LieRE · Synthetic spatial task, 276×276 | accuracy: 99.7 -> 100 | No clear effect |
| increase LieRE basis tile size from 2 to 8 (ablation) | LieRE · CIFAR100 | accuracy: 68.84 -> 70.32 | Helped |
| increase LieRE basis tile size from 2 to 8 (ablation) | LieRE · UCF101 | accuracy: 46.30 -> 47.03 | Helped |
| share LieRE parameters across heads and layers instead of learning them independently (ablation) | LieRE₈ · CIFAR-100 | accuracy: 70.3 -> 69.5 | Hurt |
| use the 2π initialization instead of 1 initialization (ablation) | LieRE₈ · CIFAR-100 | mean difference: -0.0118 | No clear effect |
Surprising findings
- Off-the-shelf multimodal LLMs (ChatGPT 4o, Claude Sonnet 3.5, Gemini Pro 1.5) were unable to solve the simple arrow-direction task.
- On UCF101 with patch shuffling, LieRE_64 shows a smaller relative drop (37.4%) than LieRE_8 (40.9%), and the 64-block variant underperforms RoPE-Mixed on UCF101 accuracy (44.7% vs 46.3%).
Limitations
- For 1D input LieRE reduces to RoPE with learnable phases, so no added capacity in 1D.
- Not directly applicable to architectures without attention such as CNNs, and encodes only vector positions in R^d, not SE(3) poses.
- Relies on the accuracy and numerical stability of PyTorch's matrix exponential.
Experimental setup
Datasets: CIFAR-100 ImageNet-1k UCF101 synthetic arrow-direction task
Benchmarks: CIFAR-100 ImageNet-1k UCF101 synthetic arrow direction task
Metrics: top-1 accuracy validation loss relative accuracy drop after patch shuffling accuracy across resolutions FLOPs
Baselines: Absolute positional encoding VisionLlama RoPE RoPE-Mixed
Builds on: RoPE RoPE-Mixed VisionLlama
Closest papers by meaning
- ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices (CVPR 2025 · similarity 0.89)
- A Circular Argument: Does RoPE need to be Equivariant for Vision? (NeurIPS 2025 · similarity 0.87)
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.86)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.86)
- What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.85)
- Learning the RoPEs: Better 2D and 3D Position Encodings with STRING (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.