Research map / Learning theory and algorithms / Prediction and decision losses
Length Optimization in Conformal Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CPL, a minimax framework that optimizes an adaptive covariate-dependent threshold over a fixed conformity score to minimize prediction set length under covariate-shift-based conditional coverage constraints.
Problem
Conformal prediction sets are often unnecessarily long, and existing methods lack a principled way to minimize length while guaranteeing conditional validity (marginal, group-conditional, or general covariate-shift classes).
Approach
Formulate length minimization under coverage constraints over a finite-dimensional affine class of covariate shifts as a minimax problem; prove strong duality in the infinite-sample regime and relax it to a learnable adaptive threshold h(x) over a fixed conformity score, optimized by gradient descent-ascent with smoothed indicators.
Key results
- On 11 real-world regression datasets with a residual score, CPL achieves length 1.68 vs 2.16 for Split Conformal at ~90% coverage; with CQR score, CQR+CPL gets 1.16 vs 1.40 for CQR+SC.
- On the synthetic group-conditional task, CPL's length approaches the numerically computed optimal oracle while achieving near-perfect group coverage, significantly shorter than BatchGCP and Split Conformal.
- On multiple-choice QA with Llama 2, CPL yields significantly smaller prediction sets than split conformal while maintaining 90% coverage (exact numbers only shown in figures).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CPL instead of LocalCP with a generic residual score | Conformal prediction on 11 real-world regression datasets · 11 real-world regression datasets | Length: 1.73 -> 1.68 | Helped |
| Use CPL instead of LocalCP with a generic residual score | Conformal prediction on 11 real-world regression datasets · 11 real-world regression datasets | Coverage: 90.09 -> 90.11 | No clear effect |
| Apply CPL to CQR scores instead of using CQR+Jackknife | CQR-based conformal prediction on 11 real-world regression datasets · 11 real-world regression datasets with CQR scores | Length: 1.32 -> 1.16 | Helped |
| Apply CPL to CQR scores instead of using CQR+Jackknife | CQR-based conformal prediction on 11 real-world regression datasets · 11 real-world regression datasets with CQR scores | Coverage: 89.78 -> 90.06 | No clear effect |
Surprising findings
- Split conformal can be far from length-optimal even for marginal coverage; the optimal marginal set can differ structurally from split conformal depending on covariate sign.
- Under a realizable hypothesis class, the relaxed minimax problem has strong duality, but the finite-sample theory does not guarantee length optimality.
Limitations
- Theory covers only finite-dimensional affine classes of covariate shifts; infinite-dimensional classes are not handled.
- Length optimality guarantees are not established in the finite-sample regime; the connection between primal and dual breaks down there.
- Finite-sample coverage guarantee holds only for stationary points of the smoothed objective with bounded hypothesis class and Lipschitz conditional score distribution.
Experimental setup
Datasets: TruthfulQA MMLU OpenBookQA PIQA BigBench WILDS RxRx1 11 real-world regression datasets synthetic regression
Benchmarks: TruthfulQA MMLU OpenBookQA PIQA BigBench RxRx1 11 regression datasets synthetic group-conditional regression
Metrics: average prediction set length marginal coverage group-conditional coverage
Baselines: Split Conformal Jackknife Local-SC LocalCP CQR BatchGCP Conditional Calibration
Builds on: split conformal prediction conformalized quantile regression conditional calibration BatchGCP
Closest papers by meaning
- Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction (ICML 2025 · similarity 0.88)
- Large language model validity via enhanced conformal prediction methods (NeurIPS 2024 · similarity 0.88)
- Hierarchical Conformal Classification (NeurIPS 2026 · similarity 0.87)
- Valid Selection among Conformal Sets (NeurIPS 2025 · similarity 0.87)
- Probabilistic Conformal Prediction with Approximate Conditional Validity (ICLR 2025 · similarity 0.87)
- Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coverage (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.