Research map / Architectures and efficiency / Attention and transformers
PaTH Attention: Position Encoding via Accumulating Householder Transformations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PaTH, a data-dependent position encoding for softmax attention built from accumulated Householder-like transformations, with a FlashAttention-style parallel training algorithm, and shows gains over RoPE and FoX on state tracking and language modeling.
Problem
RoPE's position transformation is data-independent, which limits the expressivity of transformers; they struggle with state-tracking and sequential reasoning tasks such as flip-flop language modeling and word problems.
Approach
Replace RoPE's fixed rotation with a cumulative product of data-dependent identity-plus-rank-one (Householder-like) transition matrices along the path between key and query, yielding bilinear softmax logits; combine with the Forgetting Transformer's additive forget gates as PaTH-FoX.
Key results
- On 760M models trained on 50B tokens, PaTH-FoX and PaTH average 54.2 on zero-shot commonsense tasks vs 52.7 for RoPE and 53.1 for FoX.
- On RULER at 16K context, PaTH-FoX reaches 22.6 vs 0.0 for RoPE and 4.9 for FoX; PhoneBook at 8K reaches 66.6 for PaTH-FoX vs 0.0 for PaTH and 0.0 for RoPE.
- Converting Qwen2.5-7B-Instruct to PaTH without continued pretraining yields mixed results, with degradation on some benchmarks such as HumanEval (82.32 teacher vs 77.44 student).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RoPE with PaTH | 760M Transformer · Commonsense average | accuracy: 52.7 -> 54.2 | Helped |
| Combine PaTH with FoX | 760M Transformer · Wikitext | perplexity: 18.03 -> 17.35 | Helped |
| Replace RoPE with PaTH | SmolLM2-1.7B converted model · GSM8K | score: 19.9 -> 20.1 | Helped |
| Replace RoPE with PaTH | SmolLM2-1.7B converted model · HumanEval | score: 23.1 -> 25.6 | Helped |
| Distill RoPE teacher to PaTH without continued pretraining | Qwen2.5-7B-Instruct · MMLU | score: 74.21 -> 73.28 | Hurt |
| Distill RoPE teacher to PaTH without continued pretraining | Qwen2.5-7B-Instruct · TheoremQA | score: 18.12 -> 21.88 | Helped |
| Distill RoPE teacher to PaTH without continued pretraining | Qwen2.5-7B-Instruct · GSM-8K | score: 80.29 -> 80.67 | Helped |
| Combine PaTH with FoX | 760M Transformer · RULER 16K | accuracy: 4.9 -> 22.6 | Helped |
| Combine PaTH with FoX | 760M Transformer · PhoneBook 8K | score: 17.7 -> 66.6 | Helped |
| Combine PaTH with FoX | 760M Transformer · LongBench-E 16K | score: 11.7 -> 16.1 | Helped |
Surprising findings
- Softmax over Mamba2-style forget-gated recurrence (outside-exponential placement) greatly underperformed FoX in preliminary experiments.
- Distilling an extensively trained model (Qwen2.5-7B-Instruct) without continued pretraining gives mixed results, with the PaTH student degrading on several benchmarks.
Limitations
- BF16 numerical instability of cumulative Householder products requires clipping beta, otherwise divergence can occur.
- Speed and memory overhead grows with larger head dimensions; speed comparisons only at head dim 64.
- A single reflection cannot represent rotations, which may lose RoPE's rotation-based relative position inductive bias.
Experimental setup
Datasets: Fineweb-Edu DCLM Python-Edu MegaMathWeb PG-19 CodeParrot NarrativeQA RULER BABILong PhoneBook LongBench-E flip-flop language modeling synthetic data
Benchmarks: Wikitext LAMBADA PIQA HellaSwag WinoGrande ARC-e ARC-c RULER BABILong PhoneBook LongBench-E GSM8K
Metrics: perplexity accuracy average score
Baselines: RoPE FoX Stick-Breaking Attention SmolLM2-1.7B teacher Qwen2.5-7B-Instruct teacher
Builds on: RoPE DeltaNet Forgetting Transformer FlashAttention DeltaProduct
Closest papers by meaning
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.92)
- What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.90)
- Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.90)
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.88)
- Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding (ICML 2025 · similarity 0.87)
- YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.