Research map / Large language models / Language, safety and interpretability
Do LLMs estimate uncertainty well in instruction-following?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates uncertainty estimation methods for LLM instruction-following on IFEval and a new controlled/realistic benchmark, finding existing methods near chance and probes best but weak on subtle errors.
Problem
Existing uncertainty estimation work targets factual tasks, while instruction-following uncertainty is entangled with length bias, task quality, and differing error severity across models in benchmarks like IFEval, so methods cannot be compared cleanly.
Approach
Builds a controlled benchmark with length-matched correct, completely wrong, and subtly off-target responses (plus a realistic version with natural model outputs), filtered for high task quality, and evaluates six uncertainty methods including a linear probe on hidden states.
Key results
- On IFEval, all six baseline uncertainty methods average AUROC between 0.43 and 0.53 across four models, near chance.
- Probe achieves the best AUROC in Controlled-Easy (e.g., 0.79 for LLaMA2-13B and Phi-3-mini) and Realistic (up to 0.72 for Phi-3-mini), outperforming verbalized and p(true) baselines.
- In Controlled-Hard, Probe AUROC drops below 0.60 across all models, indicating existing methods struggle with subtle instruction-following errors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sequence probability instead of perplexity | LLaMA2-chat-13B uncertainty estimation · IFEval | AUROC: 0.44 -> 0.61 | Helped |
| Use probing of internal representations instead of verbalized confidence | LLaMA2-13B uncertainty estimation · Controlled-Easy | AUROC: 0.67 -> 0.79 | Helped |
| Use probing of internal representations instead of verbalized confidence | LLaMA2-13B uncertainty estimation · Realistic | AUROC: 0.51 -> 0.64 | Helped |
| Use sequence probability instead of verbalized confidence | LLaMA2-13B uncertainty estimation · Realistic | AUROC: 0.51 -> 0.61 | Helped |
| Use probing of internal representations instead of perplexity | LLaMA2-13B uncertainty estimation · Controlled-Hard | AUROC: 0.60 -> 0.58 | No clear effect |
| Use probing of internal representations instead of perplexity | LLaMA2-7B uncertainty estimation · Controlled-Hard | AUROC: 0.57 -> 0.51 | Hurt |
| Use probing of internal representations instead of verbalized confidence | Mistral-7B uncertainty estimation · Controlled-Hard | AUROC: 0.56 -> 0.56 | No clear effect |
| Use normalized p(true) instead of perplexity | Phi-3-mini uncertainty estimation · Controlled-Hard | AUROC: 0.53 -> 0.62 | Helped |
| Replace Controlled-Easy examples with subtle off-target responses | Uncertainty estimation methods · Controlled-Hard | AUROC | Hurt |
| Use self-evaluation instead of logit-based methods | Uncertainty estimation methods · Controlled-Easy | AUROC | Helped |
| Use normalized p(true) instead of sequence probability | Uncertainty estimation methods · Realistic | AUROC | Hurt |
Surprising findings
- Smaller models (Mistral-7B, Phi-3-mini) outperform the larger LLaMA2-chat-13B on verbalized confidence.
- Punctuation instructions have the lowest success rate yet relatively high AUROC, suggesting models recognize failure despite struggling to comply.
Limitations
- Narrow scope of instruction types and domains (single simple verifiable instructions from IFEval).
- Potential data leakage since models may have seen similar tasks in pre-training.
- Relies on GPT-4 for task-quality filtering and generating controlled responses.
Experimental setup
Datasets: IFEval Controlled benchmark (GPT-4 generated) Realistic benchmark (LLM-generated responses)
Benchmarks: IFEval Controlled-Easy Controlled-Hard Realistic
Metrics: AUROC success rate
Baselines: Verbalized confidence Perplexity Sequence probability Normalized p(true) p(true) Mean token entropy Probe
Builds on: IFEval p(true) semantic entropy probing classifiers for LLM uncertainty
Closest papers by meaning
- Do LLMs ``know'' internally when they follow instructions? (ICLR 2025 · similarity 0.91)
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.91)
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.89)
- Benchmarking LLMs via Uncertainty Quantification (NeurIPS 2024 · similarity 0.89)
- Position: Uncertainty Quantification Needs Reassessment for Large Language Model Agents (ICML 2025 · similarity 0.89)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.