Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
An Information Theoretic Perspective on Conformal Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Connects conformal prediction set size to information theory, deriving upper bounds on the conditional entropy H(Y|X) via data processing and Fano inequalities, and uses them as conformal training objectives including in federated learning.
Problem
Conformal prediction set size is an intuitive uncertainty measure, but its formal relationship to established information-theoretic notions of uncertainty (such as the aleatoric conditional entropy) is unclear.
Approach
Use the data processing inequality for f-divergences and Fano's inequality (including a model-based variant) applied to the conformal prediction procedure to upper bound H(Y|X) by quantities involving the prediction set size and an arbitrary distribution Q; minimizing these bounds serves as a conformal training objective, with side information also incorporated.
Key results
- Proposed upper bounds (DPI, model-based Fano, simple Fano) used as conformal training objectives perform comparably to or better than ConfTr, with ConfTr failing to converge in some cases on CIFAR and MNIST.
- Authors state no single bound clearly wins across settings; the bounds are more robust in training than ConfTr according to the rebuttal.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the DPI or model-based Fano bound instead of ConfTr for conformal training | Conformal training on CIFAR · CIFAR | performance | Helped |
| Use the proposed bounds instead of ConfTr for conformal training | Conformal training · MNIST | training stability | Helped |
| Choose among the DPI, model-based Fano, and simple Fano bounds | The proposed bounds · Experiments | performance | No clear effect |
Surprising findings
- The KL term in the DPI bound tightens it relative to the cross-entropy alone, making the bound provably tighter than cross-entropy.
Limitations
- Simple Fano bound can be loose since it assumes a uniform, uninformative Q
- Does not decompose into epistemic and aleatoric uncertainty; only data aleatoric uncertainty H(Y|X) is addressed
- Empirically does not clearly outperform state-of-the-art conformal training
Experimental setup
Datasets: MNIST CIFAR
Benchmarks: MNIST CIFAR
Metrics: expected prediction set size (efficiency) coverage
Baselines: ConfTr ConfTr_class cross-entropy
Builds on: conformal training (ConfTr) Fano's inequality data processing inequality
Closest papers by meaning
- Robust Conformal Prediction Using Privileged Information (NeurIPS 2024 · similarity 0.90)
- Conformal Prediction as Bayesian Quadrature (ICML 2025 · similarity 0.90)
- Reliable Uncertainty Quantification in Machine Learning via Conformal Prediction (AAAI 2025 · similarity 0.89)
- Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them) (ICML 2024 · similarity 0.89)
- Direct Prediction Set Minimization via Bilevel Conformal Classifier Training (ICML 2025 · similarity 0.88)
- Conformal Prediction with Learned Features (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.