Research map / Architectures and efficiency / Attention and transformers
Rethinking Positional Encoding in Language Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BERT-R positional encoding with TUPE-R | BERT-Base · GLUE dev | Avg.: 83.66 -> 85.04 | Helped |
| Add relative positional encoding to TUPE-A | TUPE-A · GLUE dev | Avg.: 84.47 -> 85.04 | Helped |
| Use TUPE-R instead of BERT-Large positional encoding | BERT-Large · GLUE dev | Avg.: 85.82 -> 87.39 | Helped |
| Apply TUPE positional encoding to the generator and discriminator | ELECTRA-Base · GLUE dev | Avg.: 85.38 -> 86.07 | Helped |
| Add the [CLS] positional-correlation reset (ablation) | TUPE-A · GLUE dev | Avg.: 83.81 -> 84.47 | Helped |
| Add the [CLS] positional-correlation reset (ablation) | TUPE-A · CoLA | Matthews correlation: 59.46 -> 63.09 | Helped |
| Use separate word and position projection matrices (ablation) | BERT-A · GLUE dev | Avg.: 82.96 -> 83.64 | Helped |
| Remove word-to-position and position-to-word correlations (ablation) | BERT-Aᵈ · GLUE dev | Avg.: 83.64 -> 83.81 | No clear effect |
| Use a non-parametric form for positional correlation instead of the parametric form (ablation) | TUPE positional encoding · Pre-training | Convergence speed | Hurt |
| Parameterize relative position bias with projected relative-position vectors (ablation) | TUPE positional encoding · Pre-training | Model performance | No clear effect |
| Use TUPE-R at 300k rather than training BERT-A for 1M steps | BERT-Base · GLUE dev | Avg.: 82.96 -> 83.68 | Helped |
Closest papers by meaning
- Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding (ICML 2025 · similarity 0.89)
- Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation (ICML 2024 · similarity 0.87)
- Eliminating Position Bias of Language Models: A Mechanistic Approach (ICLR 2025 · similarity 0.87)
- The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.87)
- Reconsidering Positional Supervision in Masked Diffusion Language Model Training (NeurIPS 2026 · similarity 0.86)
- Wavelet-based Positional Representation for Long Context (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.