Research map / Architectures and efficiency / RNNs and normalization
Algebraic Positional Encodings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Algebraic Positional Encodings (APE), mapping the algebraic specification of sequences, trees, and grids to orthogonal matrix operators that generalize rotary encodings, and evaluates them on translation, synthetic transduction, tree tasks, and CIFAR-10.
Problem
Positional encodings for Transformers are mostly ad hoc or task-specific, lacking a principled account of how they reflect the structure of the underlying domain (sequences, trees, grids).
Approach
Interpret the group of relative paths in a domain (free group for sequences/k-ary trees, direct sums for grids) homomorphically as subgroups of the orthogonal group, so positions become orthogonal operators applied to queries and keys; RoPE emerges as a special case and tree/grid extensions follow naturally.
Key results
- On WMT14 en-de, APE with init reaches 23.93 BLEU versus 23.15 for Relative and 24.03 for frozen Rotary, comparable to the best.
- Across 13 setups, APE is the uncontested top performer in 8 and ranks among the best in 3, per the authors' summary.
- On CIFAR-10 with 300 epochs, Algebraic (grid) reaches 94.67% accuracy versus 92.79% for Sinusoidal 2D and 92.68% for Absolute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Algebraic positional encoding instead of the strongest reported baseline | Transformer base · WMT14 en→de test | BLEU: 24.03 ± 0.06 -> 23.93 ± 0.10 | Mixed |
| Add RoPE-mimicking initialization (ablation) | Algebraic positional encoding · Sequence and tree transduction tasks | perplexity | Helped |
| Add RoPE-mimicking initialization (ablation) | Algebraic (seq), Compact Convolutional Transformer · CIFAR-10, ≤150 epochs | accuracy: 88.93 ± 0.19 -> 92.68 ± 0.24 | Helped |
| Add RoPE-mimicking initialization (ablation) | Algebraic (seq), Compact Convolutional Transformer · CIFAR-10, ≤300 epochs | accuracy: 91.09 ± 0.20 -> 94.59 ± 0.15 | Helped |
| Add RoPE-mimicking initialization (ablation) | Algebraic (grid), Compact Convolutional Transformer · CIFAR-10, ≤150 epochs | accuracy: 92.95 ± 0.07 -> 93.13 ± 0.33 | No clear effect |
| Add RoPE-mimicking initialization (ablation) | Algebraic (grid), Compact Convolutional Transformer · CIFAR-10, ≤300 epochs | accuracy: 94.48 ± 0.18 -> 94.67 ± 0.06 | No clear effect |
| Use trainable sequential APE instead of tuned RoPE | Transformer · Algorithmic sequence transduction | perplexity | Mixed |
| Freeze RoPE rotation angles rather than tuning them | Transformer · Synthetic sequence transduction | perplexity | Mixed |
| Use tree-structured APE instead of sequential positional encodings | Transformer · Tree copying, breadth-first | perplexity | Hurt |
| Use tree-structured APE instead of sequential positional encodings | Transformer · Tree manipulation, breadth-first | perplexity | Mixed |
| Use grid-structured rather than sequential APE | Compact Convolutional Transformer · CIFAR-10, ≤150 epochs | accuracy: 92.68 ± 0.24 -> 93.13 ± 0.33 | Helped |
| Use grid-structured rather than sequential APE | Compact Convolutional Transformer · CIFAR-10, ≤300 epochs | accuracy: 94.59 ± 0.15 -> 94.67 ± 0.06 | No clear effect |
Surprising findings
- A sequential APE generally outperforms a trainable RoPE despite their theoretical equivalence, which the authors attribute to optimization difficulty of trigonometric functions.
- A frozen RoPE performs comparably to a randomly initialized APE in most tasks.
Limitations
- Theoretical scope limited to simple inductive groups; more complex constructions are left to future work.
- Positional encodings must be recomputed once per batch, which adds training cost that becomes evident for complete trees (linear scaling with explicit loops).
- Experimental evaluation is limited in scope, focusing on replicability and fairness rather than large-scale downstream tasks.
Experimental setup
Datasets: WMT14 en-de newstest2014 CIFAR-10 synthetic sequence copy/reverse/repeat tasks synthetic tree tasks (copy, rotate, C3 reduction, tree-ops)
Benchmarks: WMT14 en-de (newstest2014) synthetic sequence transduction (Copy, Repeat, Reverse) synthetic tree transduction (Copy, Rotate, C3, Tree-Ops) CIFAR-10
Metrics: BLEU perplexity top-1 accuracy
Baselines: Sinusoidal positional encoding Absolute positional encoding Relative positional encoding Rotary positional encoding (frozen) Rotary positional encoding (tuned) Tree-SQ Compact Convolutional Transformer
Builds on: Transformer RoPE Montague semantics
Closest papers by meaning
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.87)
- Characterizing the Expressivity of Fixed-Precision Transformer Language Models (NeurIPS 2025 · similarity 0.86)
- DAPE: Data-Adaptive Positional Encoding for Length Extrapolation (NeurIPS 2024 · similarity 0.86)
- A Formal Framework for Understanding Length Generalization in Transformers (ICLR 2025 · similarity 0.86)
- Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure (NeurIPS 2024 · similarity 0.86)
- Functional Interpolation for Relative Positions improves Long Context Transformers (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.