Research map / Large language models / Language, safety and interpretability
Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding
Yoav Gelberg, Koshi Eguchi, Takuya Akiba, Edoardo Cetin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| drop positional embeddings after pretraining and recalibrate (DroPE) | RoPE transformer 500M · Zero-shot NIAH at 2x context (multi-key) | success rate: 0.0 -> 41.6 | Improved |
| use DroPE instead of YaRN | RoPE transformer 500M · Zero-shot NIAH at 2x context (multi-key) | success rate: 0.5 -> 41.6 | Improved |
| scale RoPE with PI (ablation) | RoPE transformer 500M · Zero-shot NIAH at 2x context | success rate: 0.0 -> 0.0 | No clear effect |
| scale RoPE with RoPE-NTK (ablation) | RoPE transformer 500M · Zero-shot NIAH at 2x context (multi-query) | success rate: 0.0 -> 21.1 | Improved |
| train NoPE from scratch instead of RoPE (ablation) | 500M transformer · FineWeb training perplexity | training loss | Hurt |
| start DroPE recalibration from 14B-token RoPE checkpoint for last 2B tokens (ablation) | 500M transformer · FineWeb validation perplexity | perplexity | Improved |
| apply DroPE to SmolLM-360M | SmolLM-360M · LongBench + NIAH average | score: 2.98 -> 30.52 | Improved |
| use DroPE instead of YaRN | SmolLM-360M · LongBench + NIAH average | score: 19.94 -> 30.52 | Improved |
| use DroPE instead of RoPE-NTK on GovReport | SmolLM-360M · GovReport | score: 23.71 -> 21.87 | Hurt |
| apply DroPE to Llama2-7B with 20B-token recalibration | Llama2-7B · LongBench average | score: 20.03 -> 26.08 | Improved |
| apply DroPE to SmolLM-1.7B | SmolLM-1.7B · LongBench average | score: 3.11 -> 21.49 | Improved |
| longer recalibration budget (30B to 120B tokens) (ablation) | SmolLM-360M DroPE · six LM reasoning benchmarks | accuracy | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- YaRN — YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024) · better on 4 of 4 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.91)
- Long-Short Alignment for Effective Long-Context Modeling in LLMs (ICML 2025 · similarity 0.90)
- YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024 · similarity 0.90)
- Exploring Context Window of Large Language Models via Decomposed Positional Vectors (NeurIPS 2024 · similarity 0.90)
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.90)
- An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
