Research map / Label-efficient and robust learning / Test-time adaptation
Test-time Augmentation Improves Efficiency in Conformal Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that learning a test-time augmentation policy and aggregation weights on a held-out split and applying them before conformal scoring (APS/RAPS) shrinks prediction sets by about 10-14% on average without losing coverage.
Problem
Conformal classifiers often produce uninformatively large prediction sets, and their sets are unstable under small input perturbations.
Approach
Use test-time augmentation to transform the base classifier's probabilities before conformal scoring. Learn augmentation weights on a separate labeled split (D(TTA)) so that exchangeability with the calibration set is preserved, improving the underlying classifier without retraining.
Key results
- RAPS+TTA-Learned reduces average set size on ImageNet at alpha=0.01 from 37.75 to 31.25 with the expanded augmentation policy (about 17%).
- Across distribution shifts from ImageNet-C, TTA-Learned produces smaller prediction sets than RAPS alone with no loss of coverage.
- TTA combined with ResNet-101 yields smaller sets (26.5 classes) than ResNet-152 alone (29.6) at alpha=0.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learned TTA with the expanded augmentation policy to RAPS | ResNet-50 conformal prediction · ImageNet, α=0.01 | average prediction set size: 37.751 ± 2.334 -> 31.248 ± 2.177 | Helped |
| Add learned TTA with the expanded augmentation policy to RAPS | ResNet-50 conformal prediction · iNaturalist, α=0.01 | average prediction set size: 61.437 ± 6.067 -> 53.195 ± 4.884 | Helped |
| Add learned TTA with the expanded augmentation policy to RAPS | ResNet-50 conformal prediction · CUB-Birds, α=0.01 | average prediction set size: 15.293 ± 2.071 -> 14.045 ± 1.323 | Helped |
| Add learned TTA with the expanded augmentation policy to RAPS | ResNet-50 conformal prediction · ImageNet, α=0.05 | average prediction set size: 5.637 ± 0.357 -> 4.889 ± 0.168 | Helped |
| Add learned TTA with the expanded augmentation policy to RAPS | ResNet-50 conformal prediction · iNaturalist, α=0.05 | average prediction set size: 7.991 ± 1.521 -> 6.682 ± 0.447 | Helped |
| Add learned TTA with the expanded augmentation policy to RAPS | ResNet-50 conformal prediction · CUB-Birds, α=0.05 | average prediction set size: 3.624 ± 0.361 -> 3.571 ± 0.576 | No clear effect |
| Add learned TTA with the expanded augmentation policy to RAPS | ResNet-50 conformal prediction · ImageNet, α=0.10 | average prediction set size: 2.548 ± 0.074 -> 2.312 ± 0.054 | Helped |
| Add learned TTA with the expanded augmentation policy to RAPS | ResNet-50 conformal prediction · iNaturalist, α=0.10 | average prediction set size: 2.914 ± 0.116 -> 2.625 ± 0.043 | Helped |
| Add learned TTA with the expanded augmentation policy to RAPS | ResNet-50 conformal prediction · CUB-Birds, α=0.10 | average prediction set size: 2.038 ± 0.153 -> 1.893 ± 0.187 | No clear effect |
| Use the simple rather than expanded augmentation policy with learned weights (ablation) | RAPS+TTA-Learned on ImageNet · ImageNet, α=0.01 | average prediction set size: 31.248 ± 2.177 -> 32.702 ± 2.409 | Hurt |
| Use the simple rather than expanded augmentation policy with learned weights (ablation) | RAPS+TTA-Learned on iNaturalist · iNaturalist, α=0.01 | average prediction set size: 53.195 ± 4.884 -> 51.391 ± 5.211 | No clear effect |
| Use the simple rather than expanded augmentation policy with learned weights (ablation) | RAPS+TTA-Learned on CUB-Birds · CUB-Birds, α=0.01 | average prediction set size: 14.045 ± 1.323 -> 13.803 ± 1.734 | Helped |
Surprising findings
- TTA improves conformal efficiency mainly by raising the rank of the true class among unlikely classes, not by correcting top-1 predictions; only 3% of size-reduced sets involved a corrected Top-1 classification.
- On CUB-Birds, TTA gave little to no improvement, which the authors attribute to a small validation set.
Limitations
- Inference requires multiple forward passes and labeled data for learning the augmentation policy.
- Evaluated only on image classification.
- No gains on CUB-Birds, where the validation set is small.
Experimental setup
Datasets: ImageNet iNaturalist CUB-Birds ImageNet-C
Benchmarks: ImageNet iNaturalist CUB-Birds ImageNet-C
Metrics: average prediction set size class-conditional set size coverage size-stratified coverage violation
Baselines: RAPS APS RAPS+TTA-Avg Top-K sets ResNet-101 ResNet-152
Builds on: RAPS APS test-time augmentation split conformal prediction
Closest papers by meaning
- Backward Conformal Prediction (NeurIPS 2025 · similarity 0.89)
- Boosted Conformal Prediction Intervals (NeurIPS 2024 · similarity 0.87)
- Co-optimization for Adaptive Conformal Prediction (NeurIPS 2026 · similarity 0.87)
- CBMA: Improving Conformal Prediction through Bayesian Model Averaging (ICLR 2025 · similarity 0.86)
- Improving the Statistical Efficiency of Cross-Conformal Prediction (ICML 2025 · similarity 0.86)
- Length Optimization in Conformal Prediction (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.