Research map / Large language models / Language, safety and interpretability
Prompt Risk Control: A Rigorous Framework for Responsible Deployment of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prompt Risk Control, a framework that selects LLM prompts using high-probability upper bounds on risk measures (mean, quantile-based, and dispersion such as Gini), plus a covariate-shift extension, and validates on code generation, chatbot toxicity, and medical summarization.
Problem
Choosing a prompt by average validation performance can yield deployments with unexpectedly high losses for tail or worst-off users, and existing distribution-free risk control methods have not been applied to prompting.
Approach
Apply distribution-free uncertainty quantification (Learn Then Test, Quantile Risk Control, Statistical Dispersion Control) to prompt selection so that only prompts whose high-probability risk bound is below a threshold are returned, and extend the quantile bounds to covariate shift via importance weighting with a lower CDF bound correction.
Key results
- On MBPP with CodeLlama-7b, LTT with Hoeffding-Bentkus bounds gives tighter risk bounds than Hoeffding while remaining valid relative to held-out test loss.
- On Anthropic HH with Flan-T5-XXL, PRC selects a prompt that satisfies the 92.5% VaR toxicity constraint at alpha=0.05, whereas the best-reward prompt violates it on test data.
- Under covariate shift to Anthropic Red Team, the shifted median bound is 0.01541 and holds over the test median 0.00083, while the naive source bound of 0.00078 is violated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply the distribution-shift correction to the source-data bound | Prompt Risk Control · Anthropic Red Team target distribution | median toxicity risk bound validity: 0.00078 -> 0.01541; empirical test risk 0.00083 | Helped |
| Use Hoeffding-Bentkus instead of Hoeffding for the mean-risk bound | Prompt Risk Control on CodeLlama-7b · MBPP | mean pass@10 error risk bound | Helped |
| Select the prompt using the toxicity risk bound rather than best validation reward alone | Flan-T5-XXL chatbot · Anthropic HH | VaR toxicity at β=0.925 | Helped |
| Select a prompt subject to a Gini coefficient bound rather than choosing by best ROUGE-L alone | Falcon Instruct 40B · MeQSum | loss dispersion (Gini coefficient) | Mixed |
Surprising findings
- The naive source-only median bound (0.00078) is below the empirical test risk on the red-team target distribution, so it fails to hold.
Limitations
- PRC returns an empty set if risk constraints are unsatisfiable; weak models yield uninformative tail bounds (e.g., LLaMA 2 7B on summarization with ROUGE-L around 0.15-0.20).
- Guarantees depend on loss functions that accurately measure generation quality, and metrics like ROUGE and BLEU are insufficient.
- Guarantees require i.i.d. validation data (or covariate shift with unlabeled target data) and do not hold for arbitrary shifts or individual users.
Experimental setup
Datasets: MBPP Anthropic HH Anthropic Red Team MeQSum CNN/Daily Mail XSum
Benchmarks: MBPP Anthropic HH Anthropic Red Team MeQSum
Metrics: pass@10 VaR CVaR median loss Gini coefficient ROUGE-L toxicity reward score
Baselines: Hoeffding bound naive bound on source data
Builds on: Learn Then Test Quantile Risk Control Statistical Dispersion Control Conformal Prediction
Closest papers by meaning
- On the Worst Prompt Performance of Large Language Models (NeurIPS 2024 · similarity 0.89)
- On Prompt-Driven Safeguarding for Large Language Models (ICML 2024 · similarity 0.88)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.88)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.88)
- POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization (ICML 2025 · similarity 0.88)
- Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.