Research map / Architectures and efficiency / Attention and transformers
Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAPE, a transformer positional encoding that contextualizes positional embeddings with sequence content across layers while enforcing permutation and orthogonal (O(R)) equivariance, and shows gains in arithmetic, language modeling, and long-context retrieval.
Problem
Existing positional encodings weaken position-based addressing in transformers: they impose rigid decay patterns on attention and are learned as dataset-wide biases rather than being specialized to each instance's context.
Approach
Treat positional embeddings as tensors that are updated layer by layer by attention and MLP modules using token content, with the updates constrained to be permutation-equivariant and O(R)-equivariant so relative position relationships are preserved; RoPE is a special case and serves as initialization.
Key results
- On arithmetic addition with 2x length generalization, TAPE reaches 32.82% average accuracy vs 26.98% for FIRE and 26.32% for RoPE.
- On Proof-pile and PG19 perplexity at 8k context with Llama2 7B PEFT, TAPE reaches 2.708 and 7.063, beating LoRA, LongLoRA and Theta Scaling.
- On SCROLLS pretrained from scratch, TAPE achieves the best score on 5 of 7 tasks, e.g. QuAL EM 11.60 vs 1.25 for the next best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TAPE instead of FIRE | decoder-only transformer trained from scratch · Addition Bucket 40 (average accuracy) | accuracy: 26.98% -> 32.82% | Helped |
| Use TAPE instead of RandPE | pre-trained transformer · SCROLLS QAS | F1: 13.44 -> 11.52 | Hurt |
| Use TAPE instead of xPos | pre-trained transformer · SCROLLS CNLI | EM: 71.75 -> 72.80 | Helped |
| Use TAPE instead of xPos | pre-trained transformer · SCROLLS NQA | F1: 4.83 -> 6.79 | Helped |
| Use TAPE instead of FIRE | pre-trained transformer · SCROLLS QuAL | EM: 1.25 -> 11.60 | Helped |
| Use TAPE instead of xPos | pre-trained transformer · SCROLLS QMS | Rgm: 10.73 -> 12.42 | Helped |
| Use TAPE instead of ALiBi | pre-trained transformer · SCROLLS SumS | Rgm: 9.78 -> 10.34 | Helped |
| Use TAPE instead of ALiBi | pre-trained transformer · SCROLLS GovR | Rgm: 18.81 -> 15.18 | Hurt |
| Use TAPE instead of LoRA | Llama2 7B extended to 8192 context · Proof-pile, 8192 tokens | perplexity: 2.867 -> 2.708 | Helped |
| Use TAPE instead of Theta Scaling | Llama2 7B extended to 8192 context · PG-19, 8192 tokens | perplexity: 7.999 -> 7.063 | Helped |
Surprising findings
- Some reported baselines (e.g., FIRE on QAS at 3.41) underperform RoPE and ALiBi substantially on SCROLLS tasks, suggesting strong sensitivity to task and length.
Limitations
- Exclusive focus on decoder-only models
- Limited training scale
- Equivariance design chosen for efficiency rather than exhaustively exploring alternatives
Experimental setup
Datasets: Addition Bucket 40 C4 SCROLLS RedPajama Proof-pile PG19
Benchmarks: Addition Bucket 40 SCROLLS Proof-pile PG19 passkey retrieval
Metrics: accuracy F1 exact match Rouge geometric mean (Rgm) perplexity FLOPs throughput
Baselines: RoPE ALiBi RandPE FIRE xPos NoPE LoRA LongLoRA Theta Scaling
Builds on: RoPE FIRE T5 relative bias LoRA LongLoRA
Closest papers by meaning
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.88)
- Eliminating Position Bias of Language Models: A Mechanistic Approach (ICLR 2025 · similarity 0.88)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.87)
- Reconsidering Positional Supervision in Masked Diffusion Language Model Training (NeurIPS 2026 · similarity 0.87)
- Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs (NeurIPS 2024 · similarity 0.87)
- The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.