Research map / Large language models / Instruction and fine-tuning
An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CREAM, a positional-index manipulation recipe that fine-tunes Llama-2 at its 4K pre-trained window with continuity/relativity segmentation and truncated-Gaussian middle sampling to extend context to 256K, mitigating lost-in-the-middle.
Problem
Extending the context window of pre-trained LLMs usually requires fine-tuning at the target length, and existing position-index methods perform poorly on information located in the middle of long contexts.
Approach
Remap position indices so that long target sequences map into the pre-trained window, splitting the window into head/middle/tail segments that mix continuity (small fixed head/tail) and relativity (N/3 head/tail), and sample the middle segment's location from a truncated Gaussian to focus training on middle positions.
Key results
- CREAM-YaRN outperforms PoSE-YaRN by over 20% on average on Lost-in-the-Middle (73.4 vs 50.0 AVG at 5K tokens).
- CREAM-7B-32k reaches 35.9 macro average on LongBench, versus 34.3 for LongChat-v1.5-7B-32k, the second-best model listed.
- CREAM-Chat is fine-tuned for only 100 steps on 4K data and shows near-perfect Needle-in-a-Haystack performance up to about 29K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CREAM instead of PoSE with linear interpolation | Llama 2-7B · Lost-in-the-Middle, 5K, AVG | accuracy: 50.9 -> 65.2 | Helped |
| Use CREAM instead of PoSE with linear interpolation | Llama 2-7B · Lost-in-the-Middle, 10K, AVG | accuracy: 28.7 -> 39.4 | Helped |
| Use CREAM instead of PoSE with YaRN interpolation | Llama 2-7B · Lost-in-the-Middle, 5K, AVG | accuracy: 50.0 -> 73.4 | Helped |
| Use CREAM instead of PoSE with YaRN interpolation | Llama 2-7B · Lost-in-the-Middle, 10K, AVG | accuracy: 24.4 -> 45.2 | Helped |
| Use CREAM-7B-32k instead of LongChat-v1.5-7B-32k | Llama2-7B-Chat context extension · LongBench | AVG: 34.3 -> 35.9 | Helped |
| Use CREAM with LoRA fine-tuning instead of full-parameter fine-tuning | Llama2-7B-Chat · LongBench | Macro: 35.9 -> 35.5 | No clear effect |
| Use CREAM with QLoRA fine-tuning instead of full-parameter fine-tuning | Llama2-7B-Chat · LongBench | Macro: 35.9 -> 35.0 | No clear effect |
| Replace truncated Gaussian sampling with uniform sampling (ablation) | CREAM · LongChat-Lines retrieval | retrieval performance | Hurt |
| Remove both fixed head and tail segments (ablation) | CREAM · LongChat-Lines retrieval at 32K | score: zero score | Hurt |
| Keep only the fixed head segment (ablation) | CREAM · LongChat-Lines retrieval | retrieval performance | Hurt |
| Keep only the fixed tail segment (ablation) | CREAM · LongChat-Lines retrieval | retrieval performance | Hurt |
| Balance continuity and relativity rather than enforce only one (ablation) | CREAM · LongChat-Lines retrieval | retrieval performance | Helped |
Surprising findings
- Increasing the target context length from 64K to 256K brings little to no perplexity increase, indicating stability at extreme lengths.
- Fine-tuning Llama-3-8B on 4K data yields unexpectedly strong LongChat-Lines performance up to 32K despite its native 8K window.
Limitations
- Information loss from position interpolation remains when fine-tuning is restricted to the pre-trained length.
- Decoder-only models tend to show a U-shaped performance curve on lost-in-the-middle, so the problem is not fully solved.
- NTK interpolation gives zero results at 26K-32K on LongChat-Lines.
Experimental setup
Datasets: The Pile ShareGPT GovReport Proof-pile PG-19 Book3 LongChat-Lines Lost-in-the-Middle key-value retrieval LongBench Needle-in-a-Haystack WinoGrande TruthfulQA
Benchmarks: LongChat-Lines Lost-in-the-Middle LongBench Needle-in-a-Haystack GovReport perplexity Proof-pile perplexity PG-19 perplexity Book3 perplexity WinoGrande TruthfulQA PIQA BoolQ
Metrics: accuracy perplexity GPT score
Baselines: PoSE RandPos SkipAlign LongChat-v1.5-7B-32k Llama 2-7B LoRA QLoRA
Builds on: PoSE RandPos Positional Interpolation YaRN NTK-aware interpolation
Closest papers by meaning
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.92)
- Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.92)
- ReAttention: Training-Free Infinite Context with Finite Attention Scope (ICLR 2025 · similarity 0.90)
- PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.90)
- Exploring Context Window of Large Language Models via Decomposed Positional Vectors (NeurIPS 2024 · similarity 0.90)
- InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.