AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

On the Evaluation of Capability Estimation Methods for Large Language Models

Qiang Hu, Jin Wen, Yao Zhang, Maxime Cordy, Yongqiang Lyu

AAAI 2026 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use uncertainty-entropy feature with linear regression instead of average confidencecapability estimation · 10 datasets averageMAE: 0.2866 -> 0.0340Improved
apply temperature scaling to confidenceAverage Confidence · 10 datasets averageMAE: 0.2866 -> 0.1921Improved
combine Token Difference with Uncertainty Entropy (ablation)Token Difference feature · CosmosQAMAE: 0.0344 -> 0.0244Improved
combine two features instead of a single feature (ablation)MMB linear regression · CosmosQAMAE: 31 out of 49 cases cannot improveNo clear effect
combine Margin, Entropy and MDE uncertainty features (ablation)Uncertainty-Entropy linear regression · 10 datasets averageMAE: 0.0340 -> 0.0336Improved
add Confidence Profile to feature combination (ablation)feature combinations · 10 datasets averageMAE: 0.0336 -> 0.0462Hurt
use MLP instead of Linear Regression as meta-model (ablation)Uncertainty-Entropy · 10 datasets averageMAE: 0.0340 -> 0.7074Hurt
use KNN instead of Linear Regression as meta-model (ablation)Token Difference · 10 datasets averageMAE: 0.0357 -> 0.0442Hurt
use XGBoost instead of Linear Regression as meta-model (ablation)Uncertainty-Margin · 10 datasets averageMAE: 0.0341 -> 0.0341No clear effect
shift from in-distribution to out-of-distribution test datacapability estimation · Cosmos to Summarization / Squad to WikipediaMAE: increase in MAE of 740% and 746%Hurt
transfer from MMLU to Dialogue instead of in-distributioncapability estimation · MMLU to DialogueMAE: 0.0454 -> 0.0335Improved
use Linear Regression meta-model in OODMMB methods · OOD scenarioscount of best MAE: 20 best values vs 8 from ATCImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy