Research map / Large language models / Reasoning and chain of thought

Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data?

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EssenceBench, a coarse-to-fine benchmark compression framework that combines redundancy-based sample filtering with an iterative genetic algorithm and attribution-guided (EBM) sample selection to reconstruct LLM leaderboard scores from small subsets.

Problem

Full LLM benchmark suites are costly to run; existing compression methods (MetaBench, GAM/IRT-based selection, heuristics) either ignore sample-to-sample redundancy or search inefficiently, while still needing to preserve model rankings and score accuracy.

Approach

Combine text-embedding and ranking-correlation redundancy filtering with a genetic algorithm that uses a GAM predictor as fitness (RMSE to full-set accuracy), then iteratively refine the candidate pool using EBM-based per-sample attribution grouped into high/low/random sets.

Key results

• On GSM8K at subset size 500, EssenceBench reports RMSE 0.375 vs. 0.960 for MetaBench (about 60.7% reduction).
• On HellaSwag, EssenceBench reaches RMSE 0.863 at 200 samples versus 1.318 for MetaBench at the same size.
• Paper claims 95% of model rankings stay within 10% with about 200x fewer samples, and all shifts within 5% at roughly 400 samples on HellaSwag.

Surprising findings

• Increasing GA generations under few refinement rounds yields marginal gains or degrades performance, while more rounds consistently help.
• Attribution-based selection advantage largely disappears once subset size exceeds 400.

Limitations

• Ablation and parameter studies are reported only on GSM8K with a fixed 50-sample setting.
• Gains over MetaBench narrow at larger subset sizes (e.g. attribution benefit vanishes above 400 samples).
• Evaluation relies on Open LLM Leaderboard score matrices, so generalization to new models or benchmarks is not directly tested.

Experimental setup

Datasets: GSM8K ARC HellaSwag WinoGrande MMLU Open LLM Leaderboard

Benchmarks: GSM8K ARC HellaSwag WinoGrande MMLU

Metrics: RMSE Pearson Spearman Kendall rank stability pair accuracy NDCG@50 top-50 accuracy ranking error within 1/2/5/10%

Baselines: Random Selection GraNd PPL MetaBench

Builds on: MetaBench Generalized Additive Model Explainable Boosting Machine GraNd TinyBenchmarks

Closest papers by meaning

• How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.89)
• The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.88)
• Towards Reliable LLM Evaluation: Correcting the Winner’s Curse in Adaptive Benchmarking (NeurIPS 2026 · similarity 0.88)
• Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.87)
• Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples (NeurIPS 2025 · similarity 0.87)
• Efficient multi-prompt evaluation of LLMs (NeurIPS 2024 · similarity 0.86)