Research map / Vision-language and multimodal / Human and embodied modeling
Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bilevel Positional Encoding (BiPE), which combines absolute intra-segment encoding with relative inter-segment encoding (RoPE or ALiBi over segment indices) to improve length extrapolation in Transformer language models.
Problem
Transformers with standard positional encodings fail to generalize to sequences longer than those seen in training; the paper argues that the number of segments (sentences), not raw token length, is the main quantity to extrapolate.
Approach
Decompose each token's position into an intra-segment absolute encoding (local position within a sentence/line, added to embeddings) and an inter-segment relative encoding (segment-index distance, applied via RoPE or ALiBi in attention), motivated by the observation that per-segment token counts stay bounded as sequences grow.
Key results
- On PG-19 at 8192 length, BiPE-ALiBi achieves 25.24 perplexity vs 28.59 for ALiBi, the best baseline.
- On SCROLLS, BiPE-RoPE reaches 22.36 average score vs 18.38 for RoPE, the best baseline in that comparison.
- In arithmetic capacity experiments at hidden dim 48, BiPE-ALiBi and BiPE-RoPE reach 97% and 95% accuracy while other PEs fall below 70%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BiPE-RoPE instead of RoPE | RoPE-based model · SCROLLS Average | Average score: 18.38 -> 22.36 | Helped |
| Use BiPE-RoPE instead of RoPE | RoPE-based model · SCROLLS | Task scores | Mixed |
| Use BiPE-RoPE instead of RoPE | RoPE-based model · Normal-length benchmarks | Task-specific score | Mixed |
| Remove the intra-segment encoding (ablation) | BiPE-RoPE · PG19 | perplexity | Hurt |
| Remove the inter-segment encoding (ablation) | BiPE-RoPE · PG19 | perplexity | Hurt |
| Use fixed segment lengths instead of natural segmentation (ablation) | BiPE language model · PG19 | perplexity | Hurt |
Surprising findings
- BiPE-ALiBi gives only a small SCROLLS gain over ALiBi, and BiPE-ALiBi scores below ALiBi on several normal-length benchmarks, suggesting the benefit depends on the inter-segment encoding type.
Limitations
- Segmentation relies on symbol detection (full stop and newline), which may not apply to non-text sequences such as time series or gene sequences.
- Gains over ALiBi are modest on SCROLLS, and BiPE-ALiBi trails baselines on some normal-length benchmarks.
- Theoretical expressiveness results rely on log-precision and a constructed Transformer, not learned models.
Experimental setup
Datasets: The Pile PG-19 ArXiv GitHub SCROLLS Arithmetic dataset RACE WinoGrande TruthfulQA PIQA HellaSwag MMLU
Benchmarks: PG-19 ArXiv GitHub SCROLLS Arithmetic RACE WinoGrande TruthfulQA PIQA HellaSwag MMLU
Metrics: perplexity accuracy F1 exact match Rgm normalized accuracy
Baselines: Sinusoidal PE RoPE ALiBi XPOS Randomized RoPE RoPE_yarn
Builds on: RoPE ALiBi YaRN XPOS Randomized RoPE
Closest papers by meaning
- Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs (NeurIPS 2024 · similarity 0.87)
- Exploring Transformer Extrapolation (AAAI 2024 · similarity 0.86)
- DAPE: Data-Adaptive Positional Encoding for Length Extrapolation (NeurIPS 2024 · similarity 0.85)
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.85)
- Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding (ICML 2025 · similarity 0.85)
- Functional Interpolation for Relative Positions improves Long Context Transformers (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.