Research map / Learning theory and algorithms / Prediction and decision losses

Post-Selection Distributional Model Evaluation

NeurIPS 2026 · Learning theory and algorithms › Prediction and decision losses · approach: Adaptive testing and experiments · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PS-DME, an e-value-based framework giving distributional (CDF) confidence bands for pre-selected models with false coverage rate control under full in-sample data reuse, and shows it can be narrower than sample splitting.

Problem

Model evaluation usually certifies a single target KPI level; users need full test-time KPI distributions (performance-reliability trade-offs) for candidate models, but pre-selecting models on the same data causes post-selection bias in CDF confidence bands.

Approach

Uses e-values with an e-calibrator (power family) to invert a Kolmogorov/DKW-based test, producing CDF bands for each selected model whose false coverage rate is controlled for arbitrary data-dependent selection, using the full dataset rather than a split; gives explicit conditions under which it has narrower bands than sample splitting.</missing>

Key results

• On synthetic data with n=20, PS-DME empirical FCR is 0.0116 versus 0.0401 for SS-DME at delta=0.1, while N-PS-DME exhibits miscoverage.
• PS-DME gives uniformly lower (better) best guaranteed KPI than SS-DME across target reliability levels in the synthetic and Spider text-to-SQL experiments.
• On the O-RAN experiment, PS-DME consistently yields stronger guarantees than SS-DME variants with split ratios 0.2-0.4, except in some high-calibration regimes where SS-DME wins at 1-gamma=0.7.

Surprising findings

• Naive in-sample bands (N-PS-DME) fail to control FCR due to selection bias, as expected.
• In high-calibration regimes on O-RAN, SS-DME can outperform PS-DME at intermediate split ratios for less stringent reliability targets.

Limitations

• Guarantees apply to one-dimensional scalar KPIs; multivariate measures are left for future work.
• PS-DME width advantage depends on the split ratio and the e-calibrator parameter; sample splitting can win in some regimes.
• Experiments use small calibration sets and a proxy 'true' CDF from holdout data for validity checks in the server-log case.

Experimental setup

Datasets: Spider Colosseum O-RAN dataset HTTP server access logs (Kaggle) synthetic linear-Gaussian data

Benchmarks: Spider O-RAN buffer occupancy dataset synthetic ridge regression

Metrics: false coverage proportion false coverage rate confidence band width post-selection best guaranteed KPI

Baselines: SS-DME (sample splitting) N-PS-DME (naive in-sample)

Builds on: e-values Dvoretzky-Kiefer-Wolfowitz inequality Berk-Jones confidence bands false coverage rate (Benjamini-Yekutieli) Vovk-Sellke bound

Closest papers by meaning

• Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias (NeurIPS 2026 · similarity 0.84)
• Kernel Selection is Model Selection: A Unified Complexity-Penalised Approach for MMD Two-Sample Tests (NeurIPS 2026 · similarity 0.84)
• Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data? (ICLR 2026 · similarity 0.83)
• Unveiling the Potential of Robustness in Selecting Conditional Average Treatment Effect Estimators (NeurIPS 2024 · similarity 0.83)
• Model Assessment and Selection under Temporal Distribution Shift (ICML 2024 · similarity 0.83)
• Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation (ICLR 2024 · similarity 0.82)