Research map / Learning theory and algorithms / Prediction and decision losses

Conformal Risk Control

ICLR 2024 · Learning theory and algorithms › Prediction and decision losses · approach: Conformal prediction sets · 26 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends split conformal prediction to control the expected value of any bounded monotone loss (e.g., FNR, graph distance, F1) via a calibrated threshold, with extensions to shift, quantile, multiple, adversarial and U-risk control.

Problem

Conformal prediction only guarantees miscoverage control; many applications need control of other task-specific losses (false negative rate, graph distance, F1) in expectation for pre-trained models.

Approach

Generalize conformal prediction by choosing a threshold lambda as the infimum where a slightly inflated empirical risk (adding B/(n+1) pessimism) falls below alpha; this yields E[L_{n+1}] <= alpha for any bounded non-increasing loss, and reduces exactly to split conformal for miscoverage loss, with a matching lower bound up to 2B/(n+1).

Key results

• Tumor segmentation FNR control with PraNet at alpha=0.1 yields mean risk 0.0987 (std 0.0114) over 1000 splits.
• Hierarchical ImageNet graph-distance control with ResNet152 at alpha=0.05 yields mean risk 0.0499 (std 0.0011).
• Natural Questions F1 control with DPR at alpha=0.3 yields mean risk 0.2996 (std 0.0150) over 1000 splits.

Surprising findings

• Without monotonicity, the procedure can violate risk control by an arbitrary amount: for any epsilon there is a non-monotone loss with risk at least B - epsilon.
• Controlling risk in expectation is strictly more general than conformal coverage; the paper argues it is generally impossible to recast risk control as coverage control.

Limitations

• Non-monotone losses are not controlled in finite samples; monotonization is only powerful for near-monotone losses.
• Factor of 2 in the lower bound may not be tight (open question).
• Quantile risk control for CVaR-type risks and covariate-shift weight estimation are left open.

Experimental setup

Datasets: Kvasir Hyper-Kvasir CVC-ColonDB CVC-ClinicDB ETIS-Larib MS COCO ImageNet Natural Questions

Benchmarks: Kvasir Hyper-Kvasir CVC-ColonDB CVC-ClinicDB ETIS-Larib MS COCO ImageNet Natural Questions

Metrics: false negative rate set size graph distance loss F1 risk

Baselines: split conformal prediction high-probability risk control (Learn then Test)

Builds on: split conformal prediction weighted conformal prediction Learn then Test

Closest papers by meaning

• Conformal Risk Training: End-to-End Optimization of Conformal Risk Control (NeurIPS 2025 · similarity 0.90)
• Non-Exchangeable Conformal Risk Control (ICLR 2024 · similarity 0.89)
• Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them) (ICML 2024 · similarity 0.87)
• Occupancy-based Quantile Risk Control (NeurIPS 2026 · similarity 0.87)
• Conformal Prediction as Bayesian Quadrature (ICML 2025 · similarity 0.86)
• Online Conformal Prediction via Online Optimization (ICML 2025 · similarity 0.86)