Research map / Label-efficient and robust learning / Recognition and long-tail
Split conformal classification with unsupervised calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes split conformal classification with unsupervised calibration by assigning label weights to unlabeled calibration samples via an IPM/MMD two-sample-test objective using labeled training data, with coverage guarantees and kernel-based (RKHS) implementation.
Problem
Split conformal prediction normally needs labeled calibration samples, which are costly or impossible to obtain after training; existing weakly-supervised methods fail when all calibration samples are unlabeled.
Approach
Obtain label weights for unlabeled calibration instances by minimizing an integral probability metric (MMD for RKHS families) between weighted calibration instance-label pairs and the labeled training samples, so calibration looks statistically indistinguishable from training data; this yields a quantile of weighted conformal scores with high-probability coverage bounds governed by a bias-variance
Key results
- On 9 datasets with n=1,000 calibration samples and target coverage 0.9, unsupervised calibration gives coverage close to supervised (e.g., CIFAR10 0.91 vs 0.90; MNIST 0.90 vs 0.90), with set sizes nearly matching.
- Letter (26 classes) is the exception: unsupervised coverage 0.85 vs 0.90 supervised, indicating degradation with many classes.
- The naive pseudo-label approach reaches only 0.62-0.88 coverage, acceptable only when the classifier is highly accurate (e.g., CIFAR10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use unsupervised calibration instead of supervised calibration | Split conformal classification · CIFAR10 | coverage probability: 0.90 -> 0.91 | No clear effect |
| Use unsupervised calibration instead of supervised calibration | Split conformal classification · Satellite | coverage probability: 0.90 -> 0.90 | No clear effect |
| Use unsupervised calibration instead of supervised calibration | Split conformal classification · Letter | coverage probability: 0.90 -> 0.85 | Hurt |
| Use unsupervised calibration instead of supervised calibration | Split conformal classification · Forestcov | prediction set size: 1.90 -> 1.87 | Helped |
| Use unsupervised calibration instead of supervised calibration | Split conformal classification · CIFAR10 | prediction set size: 1.00 -> 1.01 | Hurt |
| Use unsupervised calibration instead of supervised calibration | Split conformal classification · Letter | prediction set size: 5.58 -> 4.14 | Helped |
| Use labels predicted by the classifier for calibration instead of optimized label weights | Split conformal classification · Forestcov | coverage probability: 0.90 -> 0.62 | Hurt |
| Use labels predicted by the classifier for calibration instead of optimized label weights | Split conformal classification · CIFAR10 | coverage probability: 0.90 -> 0.88 | No clear effect |
| Increase the number of unsupervised calibration samples | Proposed unsupervised calibration · CIFAR10 | coverage gap: coverage gaps smaller than 0.015 | Helped |
Surprising findings
- Naive pseudo-labeling with the classifier's predictions undercovers substantially on most datasets, even though it is intuitively simple.
Limitations
- Does not scale well with the number of classes; coverage degrades on Letter (26 classes).
- Significantly higher computational cost than conventional split conformal (quadratic/interior-point QP over nc variables).
- Coverage guarantees weaken to a bias-variance excess term G that can be large when the conformal score is non-smooth.
Experimental setup
Datasets: Drybean Forestcov Satellite Letter USPS MNIST FashionMNIST CIFAR10 ImageNet10
Benchmarks: Drybean Forestcov Satellite Letter USPS MNIST FashionMNIST CIFAR10 ImageNet10
Metrics: coverage probability prediction set size coverage gap
Baselines: supervised split conformal prediction naive unsupervised approach (classifier pseudo-labels)
Builds on: split conformal prediction adaptive prediction sets (APS) two-sample tests with IPMs
Closest papers by meaning
- Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration (NeurIPS 2024 · similarity 0.86)
- Conformal Prediction with Learned Features (ICML 2024 · similarity 0.86)
- Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning (ICLR 2025 · similarity 0.85)
- Self-Calibrating Conformal Prediction (NeurIPS 2024 · similarity 0.85)
- Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shift with Unlabeled Data (NeurIPS 2025 · similarity 0.85)
- Improving the Statistical Efficiency of Cross-Conformal Prediction (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.