Research map / Probabilistic and scientific modeling / Variational and generative sampling
Estimating Tail Risks in Language Model Output Distributions
Rico Angell, Raghav Singhal, Zachary Horvitz, Zhou Yu, Rajesh Ranganath, Kathleen McKeown, He He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sample 10,000 outputs per query instead of one greedy output | safety-tuned instruct models · STRONGREJECT 313 queries | % queries with at least one harmful output: Llama-3.1-8B: 1.6% -> 74.0%; Llama-3.2-1B: 16.6% -> 99.5% | Improved |
| estimate harm probability with importance sampling on a steered unsafe proposal instead of naive Monte Carlo | target LLMs (Llama, Qwen, Olmo, Phi-4) · STRONGREJECT | estimation error vs 10K-sample MC: matches brute force with 10-20x fewer samples (k=1000 vs 10,000) | Improved |
| use importance sampling for sycophancy and hallucination rate estimation | Qwen2.5-7B and Llama-3.1-8B · 20 safe leading queries per trait | estimation error | Improved |
| optimize proposal hyperparameters (steering intensity, switch index, mixing coefficient) with cross-entropy objective (ablation) | steered proposal model · STRONGREJECT, k=500 samples | mean squared error vs brute force | Improved |
| rewrite queries with non-adversarial paraphrases | Llama-3.1-8B, Phi-4, Qwen2.5-7B · 309 STRONGREJECT queries, 25 rewrites each | harm probability: Qwen2.5-7B: 0.0 -> 0.14 for one query | Hurt |
| forecast worst-case risk on unseen queries using extreme value theory from evaluation-set QRisk estimates | query-level estimates · STRONGREJECT paraphrases, 30% eval / 70% deployment | probability of max query risk above threshold | Improved |
Closest papers by meaning
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.88)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.88)
- Conformal Tail Risk Control for Large Language Model Alignment (ICML 2025 · similarity 0.88)
- Safety Depth in Large Language Models: A Markov Chain Perspective (NeurIPS 2025 · similarity 0.88)
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.87)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
