Research map / Large language models / Language, safety and interpretability
MetaEval: Measuring the Discrimination of Benchmarks for Efficient LLM Evaluation
Zhuo Wang, Wen Wu, Guoqing Wang, Guangze Ye, Zhenxiao Cheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select items by SD-IR discrimination vector clustering instead of correctness-pattern clustering (ablation) | MetaEval distillation · Five benchmarks | prediction error: Discrim 2.7% and Discrim+ 2.3% average error; Respons and Respons+ higher | Improved |
| add performance inference (ability estimation) on top of metaBench scores (ablation) | Discrim · Five benchmarks | prediction error: 2.7% -> 2.3% | Improved |
| use highest-discrimination items instead of near-zero-discrimination items (ablation) | ability prediction · Five benchmarks | prediction error: error reductions of 7.53%, 6.02%, 4.81%, 7.94%, 5.40% | Improved |
| use highest-discrimination bin rather than lowest (ablation) | item binning · Five benchmarks | Std / IQR of LLM accuracy: 16.72% higher Std and 29.51% higher IQR | Improved |
| Discrim vs Respons item selection on GPQA (ablation) | MetaEval · GPQA Extended (125 items) | prediction error: Respons 2.1% vs Discrim 2.7% | Hurt |
| weight items by discrimination when ranking LLMs | Open LLM Leaderboard rankings · MuSR, GPQA, BBH, ARC, MMLU-Pro | top-10 ranking: shifts up to 5 positions in MuSR | Mixed |
Closest papers by meaning
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks (ICML 2026 · similarity 0.92)
- IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation (NeurIPS 2024 · similarity 0.90)
- Truth Behind the Scene: Designing Evaluations Benchmarks to Assess LLMs’ Task-Specific Understanding over Test-Taking Strategies (AAAI 2025 · similarity 0.90)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.90)
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.90)
- Reliable and Efficient Amortized Model-based Evaluation (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
