Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Conformalized Credal Set Predictors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes inductive conformal prediction to build valid credal set predictors for classification from first-order (probabilistic) labels, using distance- and likelihood-based nonconformity scores, with validity under bounded label noise.
Problem
Learning credal set predictors (sets of distributions representing aleatoric and epistemic uncertainty) from training data labeled by probability distributions rather than single labels, with valid coverage guarantees.
Approach
Conformalize first-order probabilistic classifiers (distance-based nonconformity such as TV, WS, KL, Inner) and second-order Dirichlet predictors (1 minus relative likelihood) so that the resulting credal sets inherit conformal marginal coverage; extend to noisy multi-annotator labels via a bounded-noise assumption that shifts the threshold by epsilon.
Key results
- On ChaosNLI, the mean average test coverage across seeds aligns with or exceeds the nominal level for all nonconformity functions and miscoverage rates.
- On synthetic data with m=1 (degenerate labels), coverage is fulfilled across almost all K and m settings for TV, and the calibration quantile shrinks as m increases.
- Theorem gives coverage 1-alpha under bounded noise by adjusting alpha-tilde=(alpha-delta)/(1-delta) and raising the threshold by epsilon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply conformal credal set prediction with different nonconformity functions | ChaosNLI · ChaosNLI test set | coverage | Helped |
| use TV nonconformity with noisy probability labels | conformal credal set prediction · Synthetic data across K and m | coverage | Helped |
| increase the number of samples m used to estimate each probability distribution | conformal credal set prediction · Synthetic data | calibration nonconformity quantile | Helped |
Surprising findings
- Coverage holds even when labels are degenerate (m=1) because noisy calibration scores tend to upper-bound clean scores, yielding conservative sets.
Limitations
- No closed-form credal sets for the nonconformity functions; sets are only implicitly represented and numerical approximation is limited to few classes.
- Guarantees for noisy labels require the bounded noise assumption and a meaningful choice of epsilon and delta, which is hard in practice.
- Efficiency of credal sets for K>3 is measured via calibration quantiles rather than set size.
Experimental setup
Datasets: ChaosNLI SNLI MNLI AbductiveNLI synthetic Gaussian-softmax data
Benchmarks: ChaosNLI synthetic data
Metrics: coverage efficiency (fraction of simplex discretization within set) quantile of calibration nonconformity scores
Baselines: first-order predictor with TV nonconformity second-order Dirichlet predictor with SO nonconformity
Builds on: split conformal prediction evidential deep learning imprecise probability
Closest papers by meaning
- Credal Prediction based on Relative Likelihood (NeurIPS 2025 · similarity 0.91)
- Credal Wrapper of Model Averaging for Uncertainty Estimation in Classification (ICLR 2025 · similarity 0.91)
- Credal Learning Theory (NeurIPS 2024 · similarity 0.90)
- Credal Deep Ensembles for Uncertainty Quantification (NeurIPS 2024 · similarity 0.87)
- Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning (ICLR 2025 · similarity 0.86)
- An Information Theoretic Perspective on Conformal Prediction (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.