Research map / Large language models / Language, safety and interpretability
Mapping from Meaning: Addressing the Miscalibration of Prompt-Sensitive Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames LLM prompt sensitivity as generalization error and shows that sampling over paraphrases improves uncertainty calibration on TriviaQA/NQ, with a new embedding-variance decomposition metric that quantifies prompt sensitivity.
Problem
Black-box LLMs give very different answer distributions for semantically equivalent prompts, so the uncertainty of one prompt's output distribution miscalibrates the model's uncertainty about the prompt's meaning.
Approach
Sample answers across multiple paraphrases of a question (semantic-invariant perturbations) to approximate the response distribution over the semantic concept, and measure uncertainty with a total embedding variance metric that decomposes additively into paraphrase (epistemic) and sampling (aleatoric) uncertainty, rather than discrete entropy.
Key results
- On GPT-3.5 with m=12, the embedding-variance total metric reaches about 86.4 AUROC on TriviaQA (1,000 questions), versus 77.4 for entropy at (1,12) and ~80 for the best entropy setting.
- Paraphrase sampling (n_p=6, n_s=1) gives the best calibration for GPT-3.5 and improves calibration without reducing accuracy, unlike temperature perturbation.
- Llama 2-Chat has higher prompt-sensitivity ratio rho_u than Llama 2-Base (e.g., 52.8 vs 44.2 range on TriviaQA at comparable settings), and Base calibration is largely unchanged by perturbation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use embedding variance (total) instead of U_EigV | GPT-3.5 uncertainty calibration · TriviaQA validation, (6,2) | AUROC: 85.6$\pm$0.7 -> 86.4$\pm$0.7 | Helped |
| Use embedding variance (total) instead of U_EigV | GPT-3.5 uncertainty calibration · NQ validation, (6,2) | AUROC: 70.9$\pm$0.5 -> 72.0$\pm$0.8 | Helped |
| Spread the same 12 samples across 6 paraphrases rather than 2 (ablation) | Llama 2-Base · TriviaQA validation | AUROC: $87.8 \pm 0.6$ -> $88.1 \pm 0.7$ | Helped |
| Spread the same 12 samples across 6 paraphrases rather than 2 (ablation) | Llama 2-Chat · TriviaQA validation | AUROC: $81.8 \pm 0.5$ -> $83.6 \pm 0.5$ | Helped |
| Spread the same 12 samples across 6 paraphrases rather than 2 (ablation) | Llama 2-Base · NQ validation | AUROC: $78.3 \pm 0.2$ -> $77.9 \pm 0.4$ | No clear effect |
| Spread the same 12 samples across 6 paraphrases rather than 2 (ablation) | Llama 2-Chat · NQ validation | AUROC: $71.1 \pm 0.5$ -> $73.1 \pm 0.7$ | Helped |
| Use paraphrasing perturbations rather than no perturbation | GPT-3.5 · Question-answering tasks | AUROC | Helped |
| Use semantic-invariant perturbations rather than temperature perturbation | GPT-3.5 · Question-answering tasks | Accuracy | Helped |
| Increase paraphrase perturbations (ablation) | GPT-3.5 · TriviaQA and NQ | AUROC | Helped |
| Fine-tune Llama 2-Base with RLHF as Llama 2-Chat | Llama 2 · TriviaQA and NQ | Prompt sensitivity ratio: $52.8 \pm 0.2$ vs $53.5 \pm 0.2$; $53.4 \pm 0.3$ vs $54.6 \pm 0.2$ | Hurt |
| Increase the number of paraphrases for Llama 2-Base (ablation) | Llama 2-Base · TriviaQA and NQ | AUROC | No clear effect |
Surprising findings
- Temperature-based perturbation improves calibration only at the cost of accuracy, while semantic-invariant paraphrasing yields Pareto improvements.
- RLHF-tuned Llama 2-Chat is more prompt-sensitive than its pre-trained Base counterpart and is less well calibrated under paraphrase sampling.
Limitations
- Evaluation limited to two QA datasets with 1,000 validation questions each and three models.
- Accuracy grading relies on GPT-3.5 Turbo as judge.
- Embedding variance decomposition is evaluated only using graph-Laplacian eigenvector embeddings for QA.
Experimental setup
Datasets: TriviaQA Natural Questions (NQ)
Benchmarks: TriviaQA NQ
Metrics: AUROC (calibration) accuracy rho_u prompt sensitivity ratio
Baselines: Entropy LexiSim Semantic entropy U_EigV Random temperature sampling Fixed temperature sampling
Builds on: semantic entropy affinity graph uncertainty law of total variance uncertainty decomposition
Closest papers by meaning
- Brittlebench: Quantifying LLM robustness via prompt sensitivity (NeurIPS 2026 · similarity 0.90)
- Efficient semantic uncertainty quantification in language models via diversity-steered sampling (NeurIPS 2025 · similarity 0.89)
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.89)
- Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.89)
- Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities (NeurIPS 2024 · similarity 0.89)
- Improving Uncertainty Estimation through Semantically Diverse Language Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.