Research map / Learning theory and algorithms / Prediction and decision losses
Parametric Scaling Law of Tuning Bias in Conformal Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically and theoretically studies the coverage gap ("tuning bias") from tuning conformal prediction parameters on the same calibration data, finding it negligible for simple tuning and scaling with parameter complexity and inversely with calibration size.
Problem
Many conformal prediction methods tune parameters on a hold-out set to preserve exchangeability; the coverage impact of tuning on the same calibration set is unclear and unquantified.
Approach
Defines tuning bias as the extra coverage gap from reusing the calibration set for tuning, empirically finds it negligible for simple tuning, and bounds it via empirical process theory (DKW for finite parameter spaces, VC dimension for infinite ones), yielding a parametric scaling law.
Key results
- On CIFAR-100 with ResNet-18 (calibration size 1000), most methods (e.g., RAPS, SAPS) show negligible tuning bias, while vector scaling and ConfTr (ft.) show larger gaps when the same data is reused.
- Tuning bias of vector scaling increases by more than 50% as the number of unfrozen parameters grows, and decreases about 50% when calibration size goes from 6000 to 10000 on ImageNet.
- Theory gives finite-space bound sqrt(log(2|Lambda|)/(2n)) + ... and infinite-space bound C*sqrt((d+1)/n); temperature scaling has zero tuning bias for binary classification.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the candidate-model pool from 40 to 80 | Same-data model selection and calibration with RandomForestQuantileRegressor · Protein Structure | tuning bias: 0.0246 -> 0.0412 | Mixed |
| Increase calibration size from n=100 to n=500 | Same-data model selection and calibration with 200 candidate models · Protein Structure | tuning bias: 0.0150 -> 0.0037 | Helped |
| Use temperature scaling instead of vector scaling | ResNet-18 conformal prediction · CIFAR-100 | tuning bias: 1.13 ± 0.02 -> 0.14 ± 0.01 | Helped |
| Use temperature scaling instead of vector scaling | ResNet-18 conformal prediction · ImageNet | tuning bias: 6.63 ± 0.03 -> 0.04 ± 0.01 | Helped |
| Add order-preserving constraint to ConfTr (ft.) (ablation) | ResNet-18 conformal prediction · CIFAR-100 | tuning bias: 6.15 ± 0.86 -> 0.52 ± 0.37 | Helped |
| Add order-preserving constraint to ConfTr (ft.) (ablation) | ResNet-18 conformal prediction · ImageNet | tuning bias: 21.68 ± 0.58 -> 0.40 ± 0.31 | Helped |
| Tune with the same data rather than a separate hold-out set | RAPS and SAPS · Various settings reported in the paper | coverage gap | No clear effect |
| Tune score aggregation weights with the calibration data | Score aggregation · Various settings reported in the paper | coverage gap | No clear effect |
| Tune C-Adapter with the calibration data | C-Adapter · Various settings reported in the paper | coverage gap | No clear effect |
| Tune vector scaling with the same data rather than a hold-out set | Vector scaling · CIFAR-100 and ImageNet | coverage gap | Hurt |
| Tune ConfTr (ft.) with the same data rather than a hold-out set | ConfTr (ft.) · Except CIFAR-10 | coverage gap | Hurt |
| Increase the number of unfrozen vector-scaling parameters (ablation) | Vector scaling · ResNet-18 on ImageNet | tuning bias: increases by more than 50% | Hurt |
Surprising findings
- Tuning bias is negligible for most tuning methods (RAPS, SAPS, score aggregation, C-Adapter) even when the same data is reused for tuning and calibration.
- Order-preserving constraint on ConfTr fine-tuning sharply reduces tuning bias (e.g., 6.15% to 0.52% on CIFAR-100).
Limitations
- Derived bounds are not tight without further modeling assumptions.
- Empirical study limited to image classification with ResNet-18 and a simple CNN, plus a small regression experiment.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet Protein Structure (UCI)
Benchmarks: CIFAR-10 CIFAR-100 ImageNet Protein Structure (UCI)
Metrics: coverage gap tuning bias prediction set size
Baselines: RAPS SAPS score aggregation temperature scaling vector scaling C-Adapter ConfTr (ft.)
Builds on: split conformal prediction RAPS SAPS temperature scaling ConfTr
Closest papers by meaning
- Conformal Prediction under Lévy-Prokhorov Distribution Shifts: Robustness to Local and Global Perturbations (NeurIPS 2025 · similarity 0.88)
- Generative Conformal Prediction with Optimized Coverage Allocation (NeurIPS 2026 · similarity 0.87)
- Exploring the Noise Robustness of Online Conformal Prediction (NeurIPS 2025 · similarity 0.87)
- Leave-One-Out Stable Conformal Prediction (ICLR 2025 · similarity 0.87)
- Co-optimization for Adaptive Conformal Prediction (NeurIPS 2026 · similarity 0.87)
- Multi-model Ensemble Conformal Prediction in Dynamic Environments (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.