Research map / Learning theory and algorithms / Prediction and decision losses
Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that prediction sets are the sufficient interface for value-at-risk-averse decision makers, derives a max-min decision rule and optimal set construction, and proposes Risk-Averse Calibration (RAC) with distribution-free finite-sample coverage guarantees.
Problem
How to quantify prediction uncertainty so that risk-averse decision makers optimizing value at risk can map predictions to actions with safety guarantees and high utility.
Approach
Proves that prediction sets are optimal for value-at-risk-averse agents: the max-min decision rule over a marginally valid prediction set is minimax optimal, and the optimal sets reduce to a one-dimensional parameter beta; RAC calibrates beta via split-conformal-style search over the black-box model's probabilities.
Key results
- On medical diagnosis, RAC keeps critical-mistake rate for COVID-19 below 10% at alpha=0.02, versus over 60% for best-response, with under 5% drop in average utility.
- On MovieLens, RAC (alpha=0.05) cuts critical recommendation errors by 75% while losing about 15% average utility relative to best-response.
- RAC achieves higher worst-case utility (utility certificate) than conformal max-min baselines across all alpha levels in both experiments, and all prediction-set baselines meet target miscoverage.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RAC instead of best-response decisions | Inception_v3 medical diagnosis · COVID-19 Radiography Database | critical mistakes on COVID-19 cases: over 60% -> below 10% (at α=0.02) | Helped |
| Use RAC instead of conformal prediction baselines | Risk-averse decisions · COVID-19 Radiography Database | average realized max-min value | Helped |
| Use RAC instead of other prediction-set methods | Risk-averse decisions · COVID-19 Radiography Database | average utility | Helped |
| Use RAC instead of calibrated-model best response | Inception_v3 medical diagnosis · COVID-19 Radiography Database | average utility: under 5% drop in average utility | Mixed |
| Use RAC instead of conformal prediction baselines | Risk-averse recommendations · MovieLens | average realized max-min value | Helped |
| Use RAC instead of other CP-based methods | MovieLens recommendations · MovieLens | average utility | Helped |
| Use RAC instead of best-response decisions | MovieLens recommendations · MovieLens (α=0.05) | critical errors: 75% | Helped |
| Use RAC instead of best-response decisions | MovieLens recommendations · MovieLens (α=0.05) | average utility: 15% reduction in average utility | Hurt |
Surprising findings
- Optimal prediction sets for value-at-risk maximization need not be nested or take the form of thresholding a score function, unlike sets optimizing size.
Limitations
- Focuses on marginal rather than group-, label-, or action-conditional safety guarantees.
- Assumes the utility function is known and correctly specified; robustness to utility misspecification is untested.
- Experiments are limited to two applications with a single fixed utility matrix per task.
Experimental setup
Datasets: COVID-19 Radiography Database MovieLens
Benchmarks: COVID-19 Radiography Database MovieLens
Metrics: average realized max-min value fraction of critical mistakes average realized utility realized miscoverage
Baselines: Calibration + Best-Response (decision calibration) split conformal prediction score-1 (1 - f_x(y)) split conformal prediction score-2 (APS-style cumulative sum) greedy max-min score-3
Builds on: conformal prediction decision calibration conformal risk control nested conformal prediction
Closest papers by meaning
- Utility-Directed Conformal Prediction: A Decision-Aware Framework for Actionable Uncertainty Quantification (ICLR 2025 · similarity 0.88)
- Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets (NeurIPS 2024 · similarity 0.86)
- Designing Decision Support Systems using Counterfactual Prediction Sets (ICML 2024 · similarity 0.86)
- Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.86)
- Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them) (ICML 2024 · similarity 0.86)
- SafeAR: Safe Algorithmic Recourse by Risk-Aware Policies (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.