Research map / Learning theory and algorithms / Prediction and decision losses

SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions

NeurIPS 2026 · Learning theory and algorithms › Prediction and decision losses · approach: Conformal prediction sets · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SimplexUQ, a benchmark and protocol (SimplexTasks-12) that compares existing conformal wrappers on simplex-valued predictions, measuring coverage allocation across prediction-space strata rather than marginal coverage alone.

Problem

Split conformal prediction guarantees marginal coverage, but on simplex-valued outputs a single threshold can over-cover easy regions and under-cover hard ones; no benchmark exists for choosing among conformal wrappers for this allocation problem.

Approach

Rather than proposing a new wrapper, the paper fixes the predictor, score, and a response-free stratification map, varies only the conformal wrapper (Global, Mondrian, normalization, leave-one-out, BatchMVP, etc.), and reports marginal coverage, max disparity, worst-stratum coverage, and radius across 12 synthetic and real tasks.

Key results

• Global split CP on CIFAR-10 attains 0.900 marginal coverage but 0.542 worst-entropy-stratum coverage; Mondrian raises it to 0.886 and reduces max disparity from 0.358 to 0.022.
• Mondrian has lower disparity than Global in 23 of 24 task-map cells in the fixed-map series.
• In a 12-task comparison, Mondrian has lower default-partition disparity than BatchMVP on all 12 tasks, while BatchMVP has lower overlapping-family disparity than Mondrian on five tasks.

Surprising findings

• Global CP keeps near-nominal marginal coverage on all six real tasks while its max disparity reaches 0.490 on UTKFace.
• OneShot kNN normalization has mean marginal coverage below the 0.90 target on all six real tasks (0.882-0.892).

Limitations

• Mondrian is evaluated on the same partition it calibrates, so its advantage is relative to the chosen map.
• Some archived configurations lack explicit map provenance and several cells are not like-for-like comparisons.
• Results depend on frozen predictors, fixed scores, and exchangeability assumptions; the selection heuristic is not a validated universal rule.

Experimental setup

Datasets: CIFAR-10 20 Newsgroups SemEval AffectiveText Samson PBMC3K UTKFace SimplexTasks-12 synthetic regimes D1-D6

Benchmarks: SimplexTasks-12

Metrics: marginal coverage max disparity worst-stratum coverage mean radius runtime

Baselines: Global split CP Mondrian TwoStage OneShot TrainRes FullCP Jackknife+ Weighted BatchMVP

Builds on: split conformal prediction Mondrian conformal prediction jackknife+ BatchMVP

Closest papers by meaning

• Generative Conformal Prediction with Optimized Coverage Allocation (NeurIPS 2026 · similarity 0.88)
• Rectifying Conformity Scores for Better Conditional Coverage (ICML 2025 · similarity 0.88)
• On the Burden of Achieving Fairness in Conformal Prediction (NeurIPS 2026 · similarity 0.87)
• Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coverage (ICML 2025 · similarity 0.87)
• Stochastic Grouping Conformal Prediction for Effective Subgroup Reliability (NeurIPS 2026 · similarity 0.87)
• Exploring the Noise Robustness of Online Conformal Prediction (NeurIPS 2025 · similarity 0.86)