AI Paper Atlas

Research map / Label-efficient and robust learning / Recognition and long-tail

Improving Expert Predictions with Conformal Prediction

Eleni Straitouri, Lequn Wang, Nastaran Okati, Manuel Gomez Rodriguez

ICML 2023 · Label-efficient and robust learning › Recognition and long-tail · approach: Group-robust classification · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Require the expert to choose from calibrated conformal label sets instead of classifying unaidedExpert using conformal decision support · CIFAR-10H with ResNet-110success probability: 0.947 -> 0.987Improved
Require the expert to choose from calibrated conformal label sets instead of classifying unaidedExpert using conformal decision support · CIFAR-10H with PreResNet-110success probability: 0.947 -> 0.989Improved
Require the expert to choose from calibrated conformal label sets instead of classifying unaidedExpert using conformal decision support · CIFAR-10H with DenseNetsuccess probability: 0.947 -> 0.990Improved
Allow recommended set sizes to vary via conformal optimization rather than use top-k setsExpert using conformal decision support · CIFAR-10H with ResNet-110success probability: 0.967 -> 0.987Improved
Allow recommended set sizes to vary via conformal optimization rather than use top-k setsExpert using conformal decision support · CIFAR-10H with PreResNet-110success probability: 0.972 -> 0.989Improved
Allow recommended set sizes to vary via conformal optimization rather than use top-k setsExpert using conformal decision support · CIFAR-10H with DenseNetsuccess probability: 0.980 -> 0.990Improved
Use conformal recommendation sets and optimize their coverage parameterExpert using conformal decision support · Synthetic tasksexpert success probability: 0.3 -> 0.41; 0.5 -> 0.55; 0.7 -> 0.72; 0.9 -> 0.90Improved
Use conformal recommendation sets and optimize their coverage parameterExpert using conformal decision support · Synthetic tasksexpert success probability: 0.3 -> 0.58; 0.5 -> 0.68; 0.7 -> 0.79; 0.9 -> 0.91Improved
Increase calibration and estimation sample sizeConformal decision support · Synthetic, n = 10relative gain in empirical success probability: 47.56 ± 4.51 % -> 48.66 ± 4.54%No clear effect
Increase number of label valuesConformal decision support · Synthetic, m = 400relative gain in success probability: 48.36 ± 4.50% -> 69.44± 5.20%Improved
Use the system despite strong violations of the estimator's IIA assumptionExpert using conformal decision support · CIFAR-10Hexpert success probabilityImproved
Replace the main estimator with a difficulty-conditioned discrete choice estimatorConformal decision support · CIFAR-10Hsuccess probability: 0.947 -> 0.981; 0.947 -> 0.983; 0.947 -> 0.987Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy