Research map / Large language models / Language, safety and interpretability
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows LLM few-shot accuracy varies widely across semantically equivalent prompt formats, and proposes FormatSpread, a Thompson-sampling bandit that estimates the performance range over plausible formats without model weights.
Problem
Reported LLM performance in prompting-based evaluation typically uses a single prompt format, but meaning-preserving formatting choices may strongly change accuracy, making comparisons between models and methods unreliable.
Approach
A grammar defines plausible, semantically equivalent prompt formats; a Bernoulli Thompson-sampling bandit (FormatSpread) searches this space under an evaluation budget to estimate the minimum and maximum accuracy, yielding a performance spread without needing model weights.
Key results
- Accuracy varies by up to 76 points for LLaMA-2-13B across equivalent formats; median spread of 7.5 points across models and shot settings.
- Thompson sampling with a 51,200-evaluation budget finds a spread within 1 accuracy point of the true spread, versus 4 for naive sampling and 11 for UCB.
- Model comparison trends reverse with probability about 0.14 when only the prompt format changes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use prompt formats that omit colons after descriptors | LLaMA-2-7B on task280 · task280 | ranking accuracy: 0.043 -> 0.826 | Helped |
| Add spaces after double-colon separators | LLaMA-2-7B on task317 · task317 | ranking accuracy: 0.076 -> 0.638 | Helped |
| Replace Roman numeral option labels with letter labels | LLaMA-2-7B on task190 · task190 | ranking accuracy: 0.360 -> 0.614 | Helped |
| Remove spaces after double-colon separators | LLaMA-2-7B on task904 · task904 | ranking accuracy: 0.418 -> 0.616 | Helped |
| Replace the relevant separators with semicolons | LLaMA-2-7B on task320 · task320 | ranking accuracy: 0.361 -> 0.476 | Helped |
| Change descriptor casing from uppercase to lowercase | LLaMA-2-7B on task322 · task322 | ranking accuracy: 0.614 -> 0.714 | Helped |
| Change descriptor casing from title case to uppercase | LLaMA-2-7B on task279 · task279 | ranking accuracy: 0.372 -> 0.441 | Helped |
| Search prompt formats with Thompson sampling instead of naive sampling | Prompt spread estimation · Super-NaturalInstructions tasks | distance from true accuracy spread: within 4 points -> within 1 accuracy point of the true spread | Helped |
| Search prompt formats with Thompson sampling instead of UCB | Prompt spread estimation · Super-NaturalInstructions tasks | distance from true accuracy spread: within 11 -> within 1 accuracy point of the true spread | Helped |
| Increase model size to LLaMA-2-70B | Prompt format sensitivity · Super-NaturalInstructions, 53 tasks | accuracy spread | No clear effect |
| Increase the number of few-shot examples | Prompt format sensitivity · Super-NaturalInstructions tasks | accuracy spread | No clear effect |
| Use instruction tuning | Falcon-7B-Instruct prompt format sensitivity · Super-NaturalInstructions tasks | accuracy spread | No clear effect |
Surprising findings
- Comparison trends between models are often reversed just by changing the prompt format, so formats are not inherently good or bad.
- Format accuracy is highly non-monotonic under atomic changes, so local search methods such as simulated annealing may be ineffective.
Limitations
- Sensitivity analysis limited to classification and multiple-choice tasks with single valid outputs.
- Grammar and formats are manually constructed, so the space of plausible formats is restricted to human-style choices.
- GPT-3.5 evaluated with exact prefix matching only because full logits are unavailable.
Experimental setup
Datasets: Super-NaturalInstructions Instruction Induction
Benchmarks: Super-NaturalInstructions (53 tasks) Instruction Induction (10 tasks)
Metrics: accuracy performance spread exact prefix matching probability ranking
Baselines: UCB sampling naive sampling
Builds on: Thompson sampling upper confidence bound Super-NaturalInstructions
Closest papers by meaning
- Brittlebench: Quantifying LLM robustness via prompt sensitivity (NeurIPS 2026 · similarity 0.91)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.90)
- ReFF: Reinforcing Format Faithfulness in Language Models Across Varied Tasks (AAAI 2025 · similarity 0.90)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.90)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.90)
- On the Worst Prompt Performance of Large Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.