Research map / Large language models / Language, safety and interpretability
What is Wrong with Perplexity for Long-context Language Modeling?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper explains why perplexity fails as a long-context metric, proposes LongPPL (perplexity over key tokens identified by long-short context contrast) and LongCE (a reweighted cross-entropy loss for long-context fine-tuning).
Problem
Perplexity is widely used to evaluate long-context capability of LLMs but correlates poorly with long-context benchmark performance; the reason is unclear.
Approach
Most tokens are long-context-agnostic, so averaging PPL over all tokens obscures performance. Key tokens are identified by a causal long-short difference (LSD) plus long-context likelihood (LCL) criterion; LongPPL averages only over these tokens, and LongCE upweights them during fine-tuning.
Key results
- LongPPL (Qwen2-72B-Instruct evaluator) reaches Pearson correlation -0.96 with LongBench, -0.90 with LongEval and -0.90 with RULER, while standard PPL shows -0.18, 0.24 and 0.27.
- Key-token selection identifies LongEval answer tokens with 98.2% accuracy when combining LSD and LCL.
- LongCE improves LongEval accuracy over CE by up to 22 points (e.g., Setting A at 100 and 200 steps: 46.0 vs 24.0) and keeps non-long-context benchmark performance nearly unchanged.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace uniform CE training with LongCE token reweighting | Llama-2-7B · LongEval, PG-19 + EABF, 200 steps | accuracy: 24.0 -> 46.0; (+22.0) | Helped |
| Replace uniform CE training with LongCE token reweighting | Llama-2-7B · LongBench, PG-19 + EABF, 200 steps | average score: 26.9 -> 28.2; (+1.3) | Helped |
| Replace uniform CE training with LongCE token reweighting | Llama-2-7B · RULER, PG-19 + EABF, 200 steps | score: 42.7 -> 49.7; (+7.0) | Helped |
| Use hard key-token selection instead of standard PPL over all tokens | LongPPL · LongBench | Pearson correlation: -0.18 -> -0.96 | Helped |
| Use soft influence weights instead of hard key-token selection (ablation) | LongPPL · LongBench | Pearson correlation: -0.96 -> -0.43 | Hurt |
| Remove the LCL criterion from key-token selection (ablation) | LongPPL · Long-context benchmarks | correlation | Hurt |
| Use the evaluated model itself as the key-token evaluator (ablation) | LongPPL · Long-context benchmarks | ability to reflect model performance | Hurt |
| Select re-occurred N-grams instead of key tokens (ablation) | PPL-based long-context evaluation · LongEval | Pearson correlation: LongPPL -0.90; PPL (N-gram) -0.11 | Hurt |
| Set LongPPL thresholds to alpha=2, beta=-1 instead of the default (ablation) | LongPPL · LongEval | Pearson correlation: -0.90 -> -0.94 | Helped |
| Set LongPPL thresholds to alpha=2, beta=-1 instead of the default (ablation) | LongPPL · RULER | Pearson correlation: -0.90 -> -0.96 | Helped |
| Set LongCE short context length K=1k and sliding window length d=1k (ablation) | LongCE · LongEval, PG-19 + EABF, 200 steps | accuracy: 46.0 -> 48.0 | Helped |
| Set LongCE short context length K=1k and sliding window length d=1k (ablation) | LongCE · LongBench, PG-19 + EABF, 200 steps | average score: 28.2 -> 26.9 | Hurt |
Surprising findings
- Using the evaluated model itself as the evaluator makes LongPPL unable to distinguish models' long-context capabilities.
- PPL computed on re-occurring N-grams correlates much more weakly with long-context benchmarks than LongPPL, suggesting LongPPL is not just N-gram retrieval.
Limitations
- LongPPL with a 72B evaluator adds large computational overhead (roughly 14-20x PPL time per sequence).
- Key-token selection relies on a strong long-context evaluator model and threshold hyperparameters alpha and beta, which are tuned on a single setting.
- LongCE adds an extra forward pass, increasing training cost to about 80% more than CE by default.
Experimental setup
Datasets: GovReport LongEval LongBench RULER PG-19 Pile-arxiv MMLU ARC-Challenge RACE BigBench Hard TruthfulQA CommonsenseQA
Benchmarks: LongBench LongEval RULER Needle-in-a-Haystack MMLU ARC-Challenge RACE BigBench Hard TruthfulQA CommonsenseQA
Metrics: Pearson correlation accuracy average score perplexity
Baselines: PPL LongRecipe Standard cross-entropy (CE) loss
Builds on: position interpolation EABF perplexity
Closest papers by meaning
- A Controllable Examination for Long-Context Language Models (NeurIPS 2025 · similarity 0.91)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.89)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.89)
- NExtLong: Toward Effective Long-Context Training without Long Documents (ICML 2025 · similarity 0.89)
- HELMET: How to Evaluate Long-context Models Effectively and Thoroughly (ICLR 2025 · similarity 0.89)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.