Research map / Learning theory and algorithms / Prediction and decision losses
Direct Prediction Set Minimization via Bilevel Conformal Classifier Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPSM, a bilevel conformal training method that jointly learns a quantile via pinball-loss quantile regression (lower level) and minimizes a sigmoid-smoothed prediction set size (upper level), claiming an O(1/sqrt(n)) learning bound versus Omega(1/s) for stochastic-approximation methods.
Problem
Conformal prediction tends to yield large prediction sets; existing conformal training methods estimate the quantile on mini-batches via stochastic approximation, which incurs a learning error that scales poorly with batch size.
Approach
Formulate conformal training as bilevel optimization: the lower level learns the (1-alpha)-quantile of non-conformity scores by quantile regression (pinball loss), and the upper level minimizes a differentiable prediction-set-size loss conditioned on that learned quantile, optimized with simple first-order stochastic gradient updates on split training halves.
Key results
- DPSM reduces average prediction set size by 20.46% on average relative to the best baseline across three datasets and two architectures with HPS calibration at alpha=0.1 (per the paper).
- On Caltech-101 with DenseNet and HPS, DPSM gives 0.90 APSS vs 1.62 for CUT (44.44% reduction).
- On CIFAR-100 with APS calibration, DPSM is worse than CUT for DenseNet (2.64 vs 2.41) and ResNet on iNaturalist (79.43 vs 79.00).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DPSM instead of the strongest baseline | DenseNet and ResNet trained with HPS · CalTech-101, CIFAR-100, and iNaturalist; HPS calibration | APSS: 20.46% reduction on average across all three datasets and two scores | Helped |
| Use DPSM instead of the strongest baseline | DenseNet and ResNet trained with HPS · CalTech-101; APS calibration | APSS: 59.17%; 51.80% | Helped |
| Use DPSM instead of the strongest baseline | DenseNet and ResNet trained with HPS · CIFAR-100; APS calibration | APSS: DenseNet: 8.71% increase; ResNet: 7.35% reduction | Mixed |
| Use DPSM instead of the strongest baseline | DenseNet and ResNet trained with HPS · iNaturalist; APS calibration | APSS: DenseNet: 14.56% reduction; ResNet: 0.54% increase | Mixed |
Surprising findings
- DPSM's estimation error relative to dataset-level quantiles is initially larger than ConfTr's for the first 25 epochs, converging only later in training.
- DPSM does not beat the best baseline under APS calibration in all cases; CUT wins on CIFAR-100 DenseNet and iNaturalist ResNet.
Limitations
- Convergence guarantees for the bilevel stochastic algorithm are not provided; the authors state this is an open challenge.
- Theoretical bounds rely on assumptions (bi-Lipschitz scores, strong concavity) that are only empirically checked.
- Improvements under APS calibration are not uniform across settings.
Experimental setup
Datasets: CIFAR-100 Caltech-101 iNaturalist
Benchmarks: CIFAR-100 Caltech-101 iNaturalist
Metrics: average prediction set size marginal coverage soft set size
Baselines: CE CUT ConfTr
Builds on: ConfTr CUT conformal prediction (split CP) penalty-based bilevel optimization
Closest papers by meaning
- Optimal transport-based conformal prediction (ICML 2025 · similarity 0.89)
- Verifiably Robust Conformal Prediction (NeurIPS 2024 · similarity 0.88)
- An Information Theoretic Perspective on Conformal Prediction (NeurIPS 2024 · similarity 0.88)
- Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning (ICLR 2025 · similarity 0.88)
- Class Adaptive Conformal Training (NeurIPS 2026 · similarity 0.88)
- Hierarchical Conformal Classification (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.