Research map / Label-efficient and robust learning / Recognition and long-tail
Improving Expert Predictions with Conformal Prediction
Eleni Straitouri, Lequn Wang, Nastaran Okati, Manuel Gomez Rodriguez
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Require the expert to choose from calibrated conformal label sets instead of classifying unaided | Expert using conformal decision support · CIFAR-10H with ResNet-110 | success probability: 0.947 -> 0.987 | Improved |
| Require the expert to choose from calibrated conformal label sets instead of classifying unaided | Expert using conformal decision support · CIFAR-10H with PreResNet-110 | success probability: 0.947 -> 0.989 | Improved |
| Require the expert to choose from calibrated conformal label sets instead of classifying unaided | Expert using conformal decision support · CIFAR-10H with DenseNet | success probability: 0.947 -> 0.990 | Improved |
| Allow recommended set sizes to vary via conformal optimization rather than use top-k sets | Expert using conformal decision support · CIFAR-10H with ResNet-110 | success probability: 0.967 -> 0.987 | Improved |
| Allow recommended set sizes to vary via conformal optimization rather than use top-k sets | Expert using conformal decision support · CIFAR-10H with PreResNet-110 | success probability: 0.972 -> 0.989 | Improved |
| Allow recommended set sizes to vary via conformal optimization rather than use top-k sets | Expert using conformal decision support · CIFAR-10H with DenseNet | success probability: 0.980 -> 0.990 | Improved |
| Use conformal recommendation sets and optimize their coverage parameter | Expert using conformal decision support · Synthetic tasks | expert success probability: 0.3 -> 0.41; 0.5 -> 0.55; 0.7 -> 0.72; 0.9 -> 0.90 | Improved |
| Use conformal recommendation sets and optimize their coverage parameter | Expert using conformal decision support · Synthetic tasks | expert success probability: 0.3 -> 0.58; 0.5 -> 0.68; 0.7 -> 0.79; 0.9 -> 0.91 | Improved |
| Increase calibration and estimation sample size | Conformal decision support · Synthetic, n = 10 | relative gain in empirical success probability: 47.56 ± 4.51 % -> 48.66 ± 4.54% | No clear effect |
| Increase number of label values | Conformal decision support · Synthetic, m = 400 | relative gain in success probability: 48.36 ± 4.50% -> 69.44± 5.20% | Improved |
| Use the system despite strong violations of the estimator's IIA assumption | Expert using conformal decision support · CIFAR-10H | expert success probability | Improved |
| Replace the main estimator with a difficulty-conditioned discrete choice estimator | Conformal decision support · CIFAR-10H | success probability: 0.947 -> 0.981; 0.947 -> 0.983; 0.947 -> 0.987 | Improved |
Closest papers by meaning
- Towards Human-AI Complementarity with Prediction Sets (NeurIPS 2024 · similarity 0.91)
- Designing Decision Support Systems using Counterfactual Prediction Sets (ICML 2024 · similarity 0.90)
- Conformal Prediction Sets Improve Human Decision Making (ICML 2024 · similarity 0.88)
- Conformal Prediction for Deep Classifier via Label Ranking (ICML 2024 · similarity 0.86)
- Efficient Online Set-valued Classification with Bandit Feedback (ICML 2024 · similarity 0.86)
- Conformal Structured Prediction (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
