Research map / Probabilistic and scientific modeling / Molecules and proteins

Calibration without labels in multiple testing

NeurIPS 2026 · Probabilistic and scientific modeling › Molecules and proteins · approach: Prediction-powered and conformal inference · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that local false discovery rate estimates can be treated as approximately calibrated forecasts by building pseudo-labels from p-value spacings, enabling supervised calibration tools in multiple testing without ground-truth labels.

Problem

Multiple testing lacks observed labels for whether each null hypothesis is true, so calibration of error probabilities (such as lfdr and q-values) cannot be assessed with standard supervised tools, and common FDR-based measures may be miscalibrated under model misspecification.

Approach

Construct pseudo-labels from scaled p-value spacings (m times estimated null proportion times consecutive gaps) whose conditional mean approximates lfdr(p); this makes isotonic calibration, MLE on pseudo-labels, and empirical Bayes (Grenander) estimation equivalent, and allows reliability diagrams and Brier-regret analysis without labels.

Key results

• Isotonic lfdr estimator regret decays polynomially with m across all simulation scenarios, while p-value regret stays flat at about 0.
• In the survey data, p-values and q-values appear severely miscalibrated in reliability diagrams while the isotonic lfdr estimate lies close to the diagonal.
• Brier regret bound sqrt(2pi/m) + 2E|pi0 hat - pi0| holds without assuming the true lfdr is monotone.

Surprising findings

• The q-value can be severely miscalibrated, in some cases worse than raw p-values, on real psychology and neuroscience data.
• In simulations, q-value regret is larger than the isotonic estimator's in most cases, and in some settings the q-value decay rate is negative (regret not vanishing).

Limitations

• Theory relies on p-values being uniform under the null; extension to z-scores or high-dimensional statistics is open.
• Pseudo-label regression uses exponential errors that are exact only asymptotically and spacings are dependent.
• Analysis assumes i.i.d. two-groups model, which may not hold in practice (e.g., GWAS).

Experimental setup

Datasets: simulated two-groups p-values Szucs and Ioannidis psychology/neuroscience t-statistics (about 27,000 records)

Benchmarks: simulated two-groups model scenarios Szucs and Ioannidis survey data

Metrics: Brier regret decay rate of regret reliability diagram calibration

Baselines: raw p-values Storey q-value

Builds on: Grenander estimator pool adjacent violators algorithm Storey q-value two-groups model Dvoretzky-Kiefer-Wolfowitz inequality

Closest papers by meaning

• Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.88)
• Conformalized Multiple Testing after Data-dependent Selection (NeurIPS 2024 · similarity 0.87)
• Conditional Testing based on Localized Conformal $p$-values (ICLR 2025 · similarity 0.87)
• Finite Resources False Discovery Rate Control on Structured Hypothesis Spaces (NeurIPS 2026 · similarity 0.86)
• Exploiting LLMs for Automatic Hypothesis Assessment via a Logit-Based Calibrated Prior (NeurIPS 2025 · similarity 0.86)
• Truthfulness of Calibration Measures (NeurIPS 2024 · similarity 0.85)