Research map / Large language models / Language, safety and interpretability
On the Evaluation of Capability Estimation Methods for Large Language Models
Qiang Hu, Jin Wen, Yao Zhang, Maxime Cordy, Yongqiang Lyu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use uncertainty-entropy feature with linear regression instead of average confidence | capability estimation · 10 datasets average | MAE: 0.2866 -> 0.0340 | Improved |
| apply temperature scaling to confidence | Average Confidence · 10 datasets average | MAE: 0.2866 -> 0.1921 | Improved |
| combine Token Difference with Uncertainty Entropy (ablation) | Token Difference feature · CosmosQA | MAE: 0.0344 -> 0.0244 | Improved |
| combine two features instead of a single feature (ablation) | MMB linear regression · CosmosQA | MAE: 31 out of 49 cases cannot improve | No clear effect |
| combine Margin, Entropy and MDE uncertainty features (ablation) | Uncertainty-Entropy linear regression · 10 datasets average | MAE: 0.0340 -> 0.0336 | Improved |
| add Confidence Profile to feature combination (ablation) | feature combinations · 10 datasets average | MAE: 0.0336 -> 0.0462 | Hurt |
| use MLP instead of Linear Regression as meta-model (ablation) | Uncertainty-Entropy · 10 datasets average | MAE: 0.0340 -> 0.7074 | Hurt |
| use KNN instead of Linear Regression as meta-model (ablation) | Token Difference · 10 datasets average | MAE: 0.0357 -> 0.0442 | Hurt |
| use XGBoost instead of Linear Regression as meta-model (ablation) | Uncertainty-Margin · 10 datasets average | MAE: 0.0341 -> 0.0341 | No clear effect |
| shift from in-distribution to out-of-distribution test data | capability estimation · Cosmos to Summarization / Squad to Wikipedia | MAE: increase in MAE of 740% and 746% | Hurt |
| transfer from MMLU to Dialogue instead of in-distribution | capability estimation · MMLU to Dialogue | MAE: 0.0454 -> 0.0335 | Improved |
| use Linear Regression meta-model in OOD | MMB methods · OOD scenarios | count of best MAE: 20 best values vs 8 from ATC | Improved |
Closest papers by meaning
- Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees (NeurIPS 2025 · similarity 0.90)
- Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks (ICLR 2025 · similarity 0.89)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.87)
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.87)
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.86)
- Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
