Research map / Large language models / Language, safety and interpretability

On the Worst Prompt Performance of Large Language Models

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 16 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces RobustAlpacaEval, a benchmark of 10 semantically equivalent paraphrases per AlpacaEval query, showing large gaps between worst and best prompt performance for LLMs and that worst prompts are hard to predict or fix.

Problem

LLM performance is highly sensitive to prompt phrasing, yet prior robustness studies only vary task-level instructions on task datasets, not case-level real-world user queries; the worst-case prompt performance is unexplored.

Approach

Shift robustness evaluation from task-level instructions to case-level user queries by generating semantically equivalent paraphrases of each query, and use worst-prompt performance as the lower-bound metric; also test whether worst prompts can be predicted (perplexity, Min-K% Prob, hidden-state probes, self-preference) and whether prompt engineering or swarm distillation can raise them.

Key results

• Llama-2-70B-chat shows a 45.48-point gap between best (54.86) and worst (9.38) win-rate across 11 prompts per case.
• Scaling from Llama-2-7B to 70B raises average performance (19.52 to 29.18) but does not reduce standard deviation (13.32 to 15.61).
• Voting over paraphrases raises Llama-2-70B-chat worst performance from 9.38 to 31.36, while self-refinement and swarm distillation reduce average performance.

Measured effects reported in the paper

Surprising findings

• Worst prompts are largely model-specific: overlap of worst-1 prompts across all models is nearly zero, and prompt perplexity, Min-K% Prob, and hidden-state probes do not reliably predict prompt performance.
• Self-refinement of prompts by the model itself substantially degrades performance (e.g., Llama-2-70B average drops by 13.53 points).

Limitations

• Only ChatGPT and six open-source models from Llama, Mistral, and Gemma families were evaluated.
• No error bars or statistical significance tests reported.
• Voting improves worst-case performance at several-fold higher computational cost.

Experimental setup

Datasets: TinyAlpacaEval AlpacaEval alpaca-farm SFT set

Benchmarks: RobustAlpacaEval

Metrics: weighted win-rate worst performance best performance average performance standard deviation Kendall's W IoU of sensitive cases pairwise preference accuracy

Baselines: ChatGPT (gpt-3.5-turbo-1106) Gemma-1.1-2b-it Gemma-1.1-7b-it Mistral-7b-instruct Llama-2-7b-chat Llama-2-13b-chat Llama-2-70b-chat Self-refinement Voting

Builds on: AlpacaEval TinyAlpacaEval Swarm distillation Min-K% Prob

Closest papers by meaning

• Brittlebench: Quantifying LLM robustness via prompt sensitivity (NeurIPS 2026 · similarity 0.92)
• Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models (NeurIPS 2024 · similarity 0.90)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.90)
• Prompt Risk Control: A Rigorous Framework for Responsible Deployment of Large Language Models (ICLR 2024 · similarity 0.89)
• Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.89)
• PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries (NeurIPS 2025 · similarity 0.89)