Research map / Large language models / Instruction and fine-tuning
Large Language Models Must Be Taught to Know What They Don’t Know
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that prompting-based and sampling-based uncertainty estimates are poorly calibrated for open-ended QA, and proposes calibration tuning: LoRA fine-tuning of LLMs on a ~20k graded-answer dataset to predict answer correctness, with a user study.
Problem
Black-box uncertainty estimates (token likelihoods, verbalized or zero-shot prompted confidence, sampling-based clustering) for LLMs are poorly calibrated and expensive in open-ended generation, raising the question of how to obtain reliable correctness probabilities for LLM answers.
Approach
Fine-tune the LLM itself (via LoRA, with a JSD regularizer toward the base model) on a small dataset of its own generations labeled correct/incorrect, so that a single forward pass yields calibrated P(correct); the learned estimator also transfers to other models' answers.
Key results
- Calibration tuning with LoRA + Prompt substantially improves ECE and AUROC over zero-shot, verbalized, and sampling baselines on open-ended and multiple-choice MMLU across six base models.
- Regularization with JSD reduces mean ECE from 29.9% to 10.8% (AUROC 70.2% to 71.6%) on the ablation.
- Around 1,000 labeled examples already outperform simpler baselines, with marginal gains flattening after about 5,000 examples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with LoRA + Prompt instead of Zero-Shot Classifier | LLaMA-2-7B · LiveCodeBench LeetCode easy | ECE: 41.0% -> 46.4% | Hurt |
| Fine-tune with LoRA + Prompt instead of Zero-Shot Classifier | LLaMA-2-7B · LiveCodeBench LeetCode easy | AUROC: 56.9% -> 80.0% | Helped |
| Fine-tune with LoRA + Prompt instead of Zero-Shot Classifier | Mistral-7B · LiveCodeBench LeetCode easy | ECE: 70.2% -> 21.4% | Helped |
| Fine-tune with LoRA + Prompt instead of Zero-Shot Classifier | Mistral-7B · LiveCodeBench LeetCode easy | AUROC: 66.2% -> 85.1% | Helped |
| Fine-tune with LoRA + Prompt instead of Zero-Shot Classifier | Mistral-7B-Instruct · LiveCodeBench LeetCode easy | ECE: 52.7% -> 56.1% | Hurt |
| Fine-tune with LoRA + Prompt instead of Zero-Shot Classifier | Mistral-7B-Instruct · LiveCodeBench LeetCode easy | AUROC: 47.1% -> 70.2% | Helped |
| Add KL regularization (ablation) | Calibration-tuned models · Calibration dataset | ECE: 29.9% -> 10.8% | Helped |
| Add KL regularization (ablation) | Calibration-tuned models · Calibration dataset | AUROC: 70.2% -> 71.6% | Helped |
| Train through the model's features with LoRA + Prompt instead of using frozen Probe features | Calibration-tuned models · Open-ended MMLU | AUROC | Helped |
| Replace generated answers with incorrect answer options during training (ablation) | LoRA + Prompt · MMLU | AUROC | Hurt |
| Use frozen features with Probe instead of training through the model (ablation) | Calibration-tuned models · Open-ended MMLU | AUROC | Hurt |
Surprising findings
- A model trained only on incorrect answers still performs surprisingly well, close to a frozen probe, suggesting it partly learns question-level difficulty shortcuts.
- Mistral 7B estimates LLaMA-2 7B's correctness better than LLaMA-2 7B estimates its own, suggesting uncertainty is not model-specific.
Limitations
- Relies on two separate models for question answering and uncertainty estimation, so the approach is not a single model.
- User study is preliminary (N=181) and uses Mistral 7B Instruct on MMLU questions, which may not generalize to expert settings.
- Open-ended grading relies on GPT 3.5 Turbo, which agrees with humans less than GPT 4 does.
Experimental setup
Datasets: MMLU ARC BoolQ CommonsenseQA HellaSwag SelfAware LiveCodeBench
Benchmarks: MMLU open-ended MMLU SelfAware LiveCodeBench
Metrics: ECE AUROC
Baselines: zero-shot classifier verbalized confidence sampling counting baseline likelihood accumulation perplexity linear probe
Builds on: temperature scaling LoRA semantic uncertainty linear probes for truthfulness
Closest papers by meaning
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.92)
- Do LLMs estimate uncertainty well in instruction-following? (ICLR 2025 · similarity 0.91)
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.91)
- Benchmarking LLMs via Uncertainty Quantification (NeurIPS 2024 · similarity 0.91)
- Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function (ICML 2024 · similarity 0.90)
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.