Research map / Architectures and efficiency / Attention and transformers
Deconstructing Positional Information: From Attention Logits to Training Biases
Zihan Gu, Ruoyu Chen, Han Zhang, Hua Zhang, Yue Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace RoPE attention with MLA (NoPE+RoPE concatenation) | RoPE Transformer · Task 2 | test accuracy: 69.43 -> 97.41 | Improved |
| replace RoPE attention with MLA | RoPE Transformer · Task 1 | test accuracy: 92.64 -> 88.34 | Hurt |
| replace NoPE with RoPE | NoPE Transformer · Task 1 | test accuracy: 3.51 -> 92.64 | Improved |
| replace NoPE with RoPE | NoPE Transformer · Task 2 | test accuracy: 77.69 -> 69.43 | Hurt |
| zero out the single first-layer head (ablation) | RoPE model · Task 1 | test accuracy drop: drop of approximately 60% | Hurt |
| add absolute PE at input to RoPE model (ablation) | RoPE model · Task 1 deposit pattern | deposit pattern in layer 1: suppressed in layer 1, re-emerges in deeper layers | Mixed |
| apply RoPE to only one or two heads (rest NoPE) (ablation) | RoPE model · Task 1 | test accuracy: near-optimal with one or two RoPE heads | No clear effect |
| use MLA hybrid attention (ablation) | RoPE model · Task 1 deposit pattern | single-head dependence | Improved |
| remove absolute positional vectors vs random norm-matched vectors (ablation) | Absolute PE · layer-wise test accuracy | accuracy drop: larger drop under p_i removal than random | Improved |
Closest papers by meaning
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.92)
- Revisiting Multimodal Positional Encoding in Vision–Language Models (ICLR 2026 · similarity 0.90)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.89)
- Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding (ICML 2025 · similarity 0.88)
- What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.88)
- Wavelet-based Positional Representation for Long Context (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
