Research map / Large language models / Language, safety and interpretability
Conformal Tail Risk Control for Large Language Model Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a lightweight, training-free conformal calibration method using L-statistics to provably control distortion risk measures (CVaR, VaR) of human-rated toxicity for black-box LLM outputs.
Problem
Aligning LLM outputs so that tail risks (e.g., toxicity) measured by human ratings are controlled at a pre-specified level, when only machine disutility scores are cheap and misaligned with human ratings.
Approach
Generate nested candidate subsets filtered by a machine toxicity score, assign each subset its worst human rating, and select a single threshold via an asymptotic UCB built from L-statistics (asymptotic normality of distortion risk estimators), inverted through monotonicity to get finite-sample-style control of any distortion risk measure without model refitting.
Key results
- CDRC-L controls realized CVaR at target levels across beta in {0.5, 0.75, 0.9} and is the least conservative and least costly of the three methods (L-statistics, DKW, Berk-Jones).
- Deployment cost decreases as the human-machine Spearman correlation rises from 0.57 to 0.78 for all beta settings.
- Realized VaR under CDRC-L is extremely close to the target level, indicating little conservativeness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CDRC-L instead of CDRC-DKW or CDRC-BJ | Conformal distortion risk control for LLM toxicity · RealToxicityPrompts held-out set, CVaR control | sampling cost | Helped |
| Use CDRC-L instead of CDRC-DKW or CDRC-BJ | Conformal distortion risk control for LLM toxicity · RealToxicityPrompts held-out set, VaR control | sampling cost | Helped |
| Use CDRC-L instead of CDRC-DKW or CDRC-BJ | Conformal distortion risk control for LLM toxicity · RealToxicityPrompts held-out set, CVaR and VaR control | realized risk | Helped |
| Use Berk-Jones confidence envelopes instead of DKW envelopes | Conformal distortion risk control · RealToxicityPrompts held-out set, CVaR control | sampling cost | Helped |
| Increase human-machine score alignment | CDRC-L-calibrated LLM · RealToxicityPrompts, CVaR control | sampling cost | Helped |
| Increase human-machine score alignment | CDRC-L-calibrated LLM · RealToxicityPrompts, VaR control | sampling cost | Helped |
| Finetune Detoxify on a biased subset | Detoxify toxicity scorer · Semi-synthetic toxicity scores | ROC-AUC: 0.97 -> 0.86 | Hurt |
Limitations
- Guarantees are asymptotic (n to infinity) for the L-statistic method; finite-sample validity only for the conservative envelope baselines.
- Experiments are semi-synthetic: Detoxify serves as the human annotator, so real human-rating misalignment is not tested.
- Assumes bounded ratings and i.i.d. prompts; no guarantees under context/distribution shift.
Experimental setup
Datasets: RealToxicityPrompts Jigsaw Unintended Bias in Toxicity Classification
Benchmarks: RealToxicityPrompts (semi-synthetic toxicity setup)
Metrics: realized CVaR of human toxicity scores realized VaR average sampling cost
Baselines: CDRC-DKW CDRC-BJ
Builds on: conformal risk control L-statistics Berk-Jones statistics DKW inequality
Closest papers by meaning
- Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis (ICLR 2024 · similarity 0.88)
- Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.88)
- Your Weak LLM is Secretly a Strong Teacher for Alignment (ICLR 2025 · similarity 0.88)
- Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.88)
- Scalable Valuation of Human Feedback through Provably Robust Model Alignment (NeurIPS 2025 · similarity 0.87)
- Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.