Research map / Large language models / Language, safety and interpretability
Generalized Correctness Models: Learning Calibrated and Cross-Model Correctness Predictors from Historical Patterns
Hanqi Xiao, Vaidehi Patil, Hyunji Lee, Elias Stengel-Eskin, Mohit Bansal
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a stronger or other model as predictor instead of the target itself | correctness model · MMLU answerless | accuracy: Llama3-70B self .776 vs Qwen2.5-72B -> Llama3-70B .782 | No clear effect |
| train GCM on 8 models' histories instead of one SCM per model | Qwen3-8B correctness model · MMLU (Llama3.1-8B target) | accuracy: .792 -> .820 | Improved |
| train GCM on 8 models' histories instead of one SCM per model | Qwen3-8B correctness model · MMLU (8 target models) | AUROC: .825 -> .866 | Improved |
| add post-hoc spline calibration to GCM | GCM · TriviaQA (GCM trained on MMLU) | ECE: .105 -> .031 | Improved |
| apply GCM to held-out model Phi-3-mini | GCM · MMLU (Phi-3-mini) | accuracy: .787 -> .800 | Improved |
| apply GCM to held-out Qwen3-32B | GCM · MMLU (Qwen3-32B) | ECE: P(True) .074 -> .029 | Improved |
| remove target model name from GCM prompt (ablation) | Answer-only GCM · MMLU | accuracy: .789 -> .763 | Hurt |
| add retrieved ICL examples to verbalized confidence | Qwen3-32B predicting Gemma3-27B · MMLU | accuracy: +2.6% accuracy, -4.7% ECE | Improved |
| add retrieved ICL examples | Qwen3-8B verbalized confidence · MMLU | accuracy | No clear effect |
| train GCM without matched epochs (until overfitting) (ablation) | GCM vs SCM · MMLU (Llama3.1-8B target) | accuracy: SCM .822 vs GCM .845 | Improved |
| joint training on MMLU and TriviaQA (ablation) | GCM · MMLU (Llama3.1-8B target) | accuracy: .820 -> .827 | Improved |
| joint training on MMLU and TriviaQA (ablation) | GCM · TriviaQA (Llama3.1-8B target) | ECE: .029 -> .046 | Hurt |
Closest papers by meaning
- Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator (NeurIPS 2025 · similarity 0.89)
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.89)
- Calibrating LLMs with Semantic-level Reward (NeurIPS 2026 · similarity 0.89)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.89)
- Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only (ICLR 2025 · similarity 0.89)
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
