AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding

Yoav Gelberg, Koshi Eguchi, Takuya Akiba, Edoardo Cetin

ICLR 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
drop positional embeddings after pretraining and recalibrate (DroPE)RoPE transformer 500M · Zero-shot NIAH at 2x context (multi-key)success rate: 0.0 -> 41.6Improved
use DroPE instead of YaRNRoPE transformer 500M · Zero-shot NIAH at 2x context (multi-key)success rate: 0.5 -> 41.6Improved
scale RoPE with PI (ablation)RoPE transformer 500M · Zero-shot NIAH at 2x contextsuccess rate: 0.0 -> 0.0No clear effect
scale RoPE with RoPE-NTK (ablation)RoPE transformer 500M · Zero-shot NIAH at 2x context (multi-query)success rate: 0.0 -> 21.1Improved
train NoPE from scratch instead of RoPE (ablation)500M transformer · FineWeb training perplexitytraining lossHurt
start DroPE recalibration from 14B-token RoPE checkpoint for last 2B tokens (ablation)500M transformer · FineWeb validation perplexityperplexityImproved
apply DroPE to SmolLM-360MSmolLM-360M · LongBench + NIAH averagescore: 2.98 -> 30.52Improved
use DroPE instead of YaRNSmolLM-360M · LongBench + NIAH averagescore: 19.94 -> 30.52Improved
use DroPE instead of RoPE-NTK on GovReportSmolLM-360M · GovReportscore: 23.71 -> 21.87Hurt
apply DroPE to Llama2-7B with 20B-token recalibrationLlama2-7B · LongBench averagescore: 20.03 -> 26.08Improved
apply DroPE to SmolLM-1.7BSmolLM-1.7B · LongBench averagescore: 3.11 -> 21.49Improved
longer recalibration budget (30B to 120B tokens) (ablation)SmolLM-360M DroPE · six LM reasoning benchmarksaccuracyImproved

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy