Research map / Large language models / Language, safety and interpretability
HELMET: How to Evaluate Long-context Models Effectively and Thoroughly
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HELMET, an application-centric long-context benchmark (seven categories, inputs up to 128K) with model-based metrics and few-shot prompting, and uses it to evaluate 59 long-context LMs.
Problem
Long-context language model evaluation relies on synthetic NIAH tasks or arbitrary dataset subsets, which are noisy and inconsistent, with insufficient coverage, short lengths, unreliable ROUGE metrics, and incompatibility with base models.
Approach
A unified benchmark spanning RAG, citation generation, re-ranking, many-shot ICL, long-document QA, summarization, and synthetic recall, with controllable lengths up to 128K, GPT-4o reference-based judging for QA and summarization, and 2-shot prompting so base models can be evaluated.
Key results
- Synthetic tasks such as NIAH are weak predictors of downstream performance; no synthetic task averages Spearman correlation above 0.8 with real-world tasks across 35 instruction-tuned models.
- Closed-source models (e.g., GPT-4o-08, Gemini-1.5-Pro) lead, with generation with citations and re-ranking gaps of 30 to 40 absolute points over the best open-source models at 128K.
- Model-based summarization metric reached Cohen's kappa of 0.91 (precision) and 0.76 (recall) with human annotators on InfiniteBench Sum.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NIAH as a proxy for downstream long-context performance | long-context language models · Downstream tasks | Spearman rank correlation: all correlations are ≤0.8 | Hurt |
| Use the RULER average score as a proxy for downstream performance | long-context language models · Downstream tasks | Spearman rank correlation: all <0.85 | Hurt |
| Use HotpotQA RAG results as a proxy for other real-world tasks | long-context language models · Real-world tasks | Correlation | Helped |
| Replace ROUGE-L with reference-based model evaluation | long-document QA and summarization evaluation · Summarization | Alignment with human judgment: Cohen's κ = 0.91 for summary precision and κ = 0.76 for recall | Helped |
| Add 2-shot demonstrations (ablation) | Llama-3.1-8B base · JSON KV, 128K | SubEM: $77.3_{2.1}$ -> $98.0_{1.0}$ | Helped |
| Add 2-shot demonstrations (ablation) | Llama-3.1-8B base · MSMARCO, 128K | NDCG@10: $0.1_{0.0}$ -> $7.5_{0.5}$ | Helped |
| Add 2-shot demonstrations (ablation) | Llama-3.2-3B base · JSON KV, 128K | SubEM: $30.3_{2.1}$ -> $54.0_{8.9}$ | Helped |
| Add 2-shot demonstrations (ablation) | Llama-3.2-3B base · MSMARCO, 128K | NDCG@10: $0.1_{0.0}$ -> $6.0_{1.3}$ | Helped |
| Evaluate on diverse application-centric categories rather than a single category | long-context language model evaluation · HELMET categories | Cross-category correlation | Mixed |
| Increase context length | frontier long-context models · Generation with citations | Performance | Mixed |
Surprising findings
- Llama-3.2 models score near zero on InfiniteBench at 128K but rank well on HELMET, suggesting prior benchmark results underrate them due to prompting/format issues.
- Many open-source models outperform closed-source models on in-context learning, possibly because heavy instruction tuning hurts ICL.
Limitations
- Model-based evaluation relies on GPT-4o as judge, with human agreement only checked on small samples (10 summaries per dataset).
- Evaluation uses 100 to 600 sampled examples per dataset, and ICL uses 500, which may limit statistical precision.
- Lack of large-scale human studies on LCLM rankings makes true rankings hard to verify.
Experimental setup
Datasets: Natural Questions TriviaQA HotpotQA PopQA ALCE ASQA ALCE QAMPARI MS MARCO TREC Coarse TREC Fine NLU BANKING77 CLINC150
Benchmarks: HELMET NIAH RULER InfiniteBench ZeroSCROLLS
Metrics: SubEM NDCG@10 accuracy ROUGE F1 model-based score Spearman correlation Cohen's kappa
Baselines: NIAH RULER InfiniteBench ZeroSCROLLS LongBench L-Eval
Builds on: RULER InfiniteBench ALCE L-Eval ZeroSCROLLS
Closest papers by meaning
- A Controllable Examination for Long-Context Language Models (NeurIPS 2025 · similarity 0.91)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
- Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack (NeurIPS 2024 · similarity 0.89)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.89)
- What is Wrong with Perplexity for Long-context Language Modeling? (ICLR 2025 · similarity 0.89)
- LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.