Research map / Learning theory and algorithms / Prediction and decision losses
Conformal prediction sets
Construct prediction sets with coverage guarantees and improved efficiency or conditional validity.
116 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Prediction and decision losses, keyword cluster “conformal · predictions · online”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
16 papers in 2023–24 and 90 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 3.0% to 15.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Conformal Surface Alignment With Optimal Mobius Search (CVPR 2016 · 10 citations)
- 2019 · Conformalized Quantile Regression (NeurIPS 2019 · 49 citations)
- 2021 · Exact Optimization of Conformal Predictors via Incremental and Decremental Learning (ICML 2021 · 4 citations)
- 2022 · Learning Optimal Conformal Classifiers (ICLR 2022 · 11 citations)
- 2023 · Approximating Full Conformal Prediction at Scale via Influence Functions (AAAI 2023 · 4 citations)
- 2024 · Conformal Risk Control (ICLR 2024 · 26 citations)
- 2025 · Conformal Thresholded Intervals for Efficient Regression (AAAI 2025 · 6 citations)
- 2026 · Conditional Coverage Diagnostics for Conformal Prediction (ICML 2026 · 1 citation)
Earliest papers here
- Conformal Surface Alignment With Optimal Mobius Search (CVPR 2016 · 10 citations)
- Conformalized Quantile Regression (NeurIPS 2019 · 49 citations)
- Computing Full Conformal Prediction Set with Approximate Homotopy (NeurIPS 2019 · 4 citations)
What has been tried
56 papers here report 492 measured changes (a component added, removed or swapped, compared with a baseline): helped 61%, no clear effect 13%, hurt 19%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| increase average conformal set size from 2.49 to 2.77 | GoEmotions · human accuracy | 59.1±1.2% -> 56.9±1.3% | Hurt | Conformal Prediction Sets Improve Human Decision Making (ICML 2024 · 8 citations) |
| Remove the labeling-guided selection and use a small clean reference set | Visual datasets, α=0.03, contamination 1% · Power (relative to Standard) | 1.041 (± 0.0177) -> 0.714 (± 0.0448) | Hurt | Robust Conformal Outlier Detection under Contaminated Reference Data (ICML 2025 · 1 citation) |
| Increase ε | Question answering on SQuAD · coverage rate | Hurt | Conformal Structured Prediction (ICLR 2025 · 1 citation) | |
| use classwise lambdas | ACSIncome · disparate impact | 0.815 -> 0.797 | Hurt | A Generic Framework for Conformal Fairness (ICLR 2025 · 1 citation) |
| add change-point-triggered learning-rate resets | M4 time series · variance | 1.320579 -> 1.144337 | Helped | Online conformal prediction with decaying step sizes (ICML 2024 · 1 citation) |
| Use harmonic-mean aggregation rather than arithmetic mean, geometric mean, or quantile aggregation | Appendix C aggregation comparison · interval length | Helped | False Coverage Proportion Control for Conformal Prediction (ICML 2025 · 0 citations) | |
| Add order-preserving constraint to ConfTr (ft.) | CIFAR-100 · tuning bias | 6.15 ± 0.86 -> 0.52 ± 0.37 | Helped | Parametric Scaling Law of Tuning Bias in Conformal Prediction (ICML 2025 · 0 citations) |
| Use k=5 instead of k=1 intervals | Simulated data (1D features) · average volume | 1.14 -> 0.45 | Helped | Volume Optimality in Conformal Prediction with Structured Prediction Sets (ICML 2025 · 0 citations) |
| Replace Mean Imputation with kNN or MICE | Simulation · coverage | No clear effect | Conformal Prediction with Cellwise Outliers: A Detect-then-Impute Approach (ICML 2025 · 0 citations) | |
| use the quantile envelope instead of the linear envelope | Synthetic data, m=500 · relative length | Helped | Transductive Conformal Inference for Full Ranking (NeurIPS 2025 · 0 citations) | |
| Use (f, U) auxiliary score distributions instead of (min, max) | iWildCam · coverage | increased coverage by more than 7% on average; improved coverage by almost 10% | Helped | Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shift with Unlabeled Data (NeurIPS 2025 · 0 citations) |
| Use APS score instead of TPS | CIFAR-10, radius 0.06, coverage 0.90 · average set size | 2.70 ± 0.03 -> 2.98 ± 0.02 | Hurt | One Sample is Enough to Make Conformal Prediction Robust (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 61 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 11 of 61 papers
- CIFAR-100 · 7 of 61 papers
- MNIST · 7 of 61 papers
- ImageNet · 6 of 61 papers
- bio · 4 of 61 papers
- blog · 4 of 61 papers
- bike · 3 of 61 papers
- ACSIncome · 3 of 61 papers
Baselines
- CQR · 9 of 61 papers
- CHR · 6 of 61 papers
- PCP · 5 of 61 papers
- RAPS · 4 of 61 papers
- vanilla conformal prediction · 4 of 61 papers
- split conformal prediction · 3 of 61 papers
- ACI · 3 of 61 papers
- APS · 3 of 61 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 106 papers; from OpenAlex and the papers' affiliations).
- Stanford University · 5 papers
- University of Pennsylvania · 5 papers
- Inria · 4 papers
- Carnegie Mellon University · 4 papers
- ETH Zurich · 3 papers
- City University of Hong Kong · 3 papers
- University of California, Berkeley · 3 papers
- Shanghai Jiao Tong University · 3 papers
- Alpha Benito Research · 2 papers
- Massachusetts Institute of Technology · 2 papers
Use it
Download BibTeX of 33 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Algorithms with explicit predictors (62 papers)
- Probabilistic calibration (55 papers)
Elsewhere in the topic
- Adaptive testing and experiments (201 papers)
- Active learning and subset selection (199 papers)
- Distributionally robust optimization (184 papers)
- Data attribution and recourse (160 papers)
- Theory of neural network sensitivity (143 papers)
- Ranking models from comparisons (127 papers)
- Determinantal point process inference (122 papers)
- Provable post-hoc explanations (92 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).