Research map / Large language models / Instruction and fine-tuning
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PoSE, a fine-tuning method that extends LLM context windows by simulating long inputs within a fixed 2k window via skipping position-index biases per chunk, reaching 128k for LLaMA with far lower training cost than full-length fine-tuning.
Problem
Extending the context window of pre-trained RoPE-based LLMs normally requires fine-tuning on inputs at the full target length, whose memory and time cost grows quadratically with length.
Approach
Split the original context window into chunks and add a randomly sampled skipping bias to each chunk's position indices (re-sampled per example), so the model sees relative positions spanning the target length while training only at the original length; within-chunk positions stay continuous to resemble pre-training.
Key results
- PoSE with 2k training window reaches perplexity close to Full-length fine-tuning at 16k on GovReport (4.60 vs 4.59) and Proof-pile (2.60 vs 2.53) with Linear interpolation.
- PoSE-extended 16k/32k models maintain at least 90% passkey retrieval accuracy within their target windows, while Original, PI-only and RandPos drop to 0 beyond 2k.
- LLaMA extended to 128k via PoSE with YaRN interpolation achieves perplexity 11.33 on PG-19 at 128k, and PoSE-YaRN-128k keeps most of the original benchmark performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PoSE with a 2k training window instead of full-length fine-tuning | LLaMA-7B · GovReport, 16k evaluation | perplexity: 4.59 -> 4.60 | No clear effect |
| Use PoSE with a 2k training window instead of full-length fine-tuning | LLaMA-7B · Proof-pile, 16k evaluation | perplexity: 2.53 -> 2.60 | No clear effect |
| Train with PoSE instead of RandPos | LLaMA-7B · GovReport, 16k evaluation; 16k target | perplexity: 15.16 -> 4.60 | Helped |
| Train with PoSE instead of RandPos | LLaMA-7B · Proof-pile, 16k evaluation; 16k target | perplexity: 7.73 -> 2.60 | Helped |
| Use YaRN interpolation rather than Linear interpolation | PoSE-128k · PG-19, 128k evaluation | perplexity: 31.18 -> 11.33 | Helped |
| Use YaRN interpolation rather than Linear interpolation | PoSE-128k · Books3, 128k evaluation | perplexity: 70.87 -> 13.81 | Helped |
| Set chunk content offset v_i=0 instead of sampling v_i (ablation) | PoSE · GovReport, 16k evaluation | perplexity: 4.60 -> 4.68 | No clear effect |
| Set chunk content offset v_i=0 instead of sampling v_i (ablation) | PoSE · Proof-pile, 16k evaluation | perplexity: 2.60 -> 2.61 | No clear effect |
| Set chunk content offset v_i=u_i instead of sampling v_i (ablation) | PoSE · GovReport, 16k evaluation | perplexity: 4.60 -> 4.60 | No clear effect |
| Set chunk content offset v_i=u_i instead of sampling v_i (ablation) | PoSE · Proof-pile, 16k evaluation | perplexity: 2.60 -> 2.56 | No clear effect |
| Increase chunk number from 2 to 3 (ablation) | PoSE · Proof-pile, 16k evaluation | perplexity: 2.60 -> 2.59 | Helped |
| Increase chunk number from 2 to 2048 (ablation) | PoSE · Proof-pile, 16k evaluation | perplexity: 2.60 -> 7.73 | Hurt |
Surprising findings
- RandPos with 2048 chunks (a near-random set of position indices) performs much worse than PoSE with 2 chunks, showing that too many chunks harm performance.
- Linear interpolation at 128k degrades benchmark accuracy noticeably (e.g., ARC-C 39.93 vs 51.19 original), whereas YaRN keeps performance close to the original.
Limitations
- Full-length comparison was not run beyond 16k because V100 hardware could not afford full fine-tuning at 32k or above.
- Perplexity and performance degrade as the supported context length grows, and linear interpolation at 128k hurts benchmark accuracy.
- Chunk number is a trade-off: too many chunks deviate from pre-training position structure and hurt performance.
Experimental setup
Datasets: The Pile GovReport Proof-pile Books3 PG-19
Benchmarks: GovReport Proof-pile PG-19 Books3 passkey retrieval BoolQ PIQA WinoGrande TruthfulQA ARC-C HellaSwag
Metrics: perplexity retrieval accuracy accuracy mc2 normalized accuracy
Baselines: Full-length fine-tuning RandPos PI-only Original LLaMA
Builds on: Position Interpolation RandPos NTK-aware interpolation YaRN RoPE
Closest papers by meaning
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.91)
- Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.91)
- LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning (ICML 2024 · similarity 0.90)
- An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding (NeurIPS 2024 · similarity 0.90)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
- YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.