Research map / Learning theory and algorithms / Prediction and decision losses
Zipper: Addressing Degeneracy in Algorithm-Agnostic Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Zipper, a cross-fitted goodness-of-fit test that overlaps two testing splits with a slider parameter tau to avoid degeneracy under the null while retaining power.
Problem
Predictiveness-comparison test statistics for algorithm-agnostic goodness-of-fit testing degenerate under the null hypothesis of equal predictiveness, so the standardized statistic does not converge to a non-degenerate normal distribution, invalidating standard inference.
Approach
Instead of splitting the test fold into two disjoint halves (vanilla additional splitting) or using the same fold (degenerate), the test fold is split into two halves that overlap by a proportion tau; the overlap is weighted within the predictiveness estimates, yielding an asymptotically normal statistic under H0 for any fixed tau in [0,1) with power that increases in tau.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Zipper with overlapping testing splits rather than the non-overlapping split baselines | Variable-importance tests · Normal and binomial null simulations, n=500, p=5 or 1000 | empirical size (%): Zipper: 3.9, 4.3, 3.7, 5.6 | No clear effect |
| use Zipper rather than WGSC-2 | Variable-importance tests · Normal and binomial null simulations, n=500, p=1000 | empirical size (%): 16.7 -> 4.3; 19.9 -> 5.6 | Helped |
| use Zipper rather than DSP-Pert | Variable-importance tests · Normal and binomial null simulations, n=500, p=1000 | empirical size (%): 35.0 -> 4.3; 38.6 -> 5.6 | Helped |
| increase slider τ from 0 to 0.9 (ablation) | Zipper · Normal, p=5, δ=2, n=500 | empirical power (%): 46.9 -> 100.0 | Helped |
| increase slider τ from 0 to 0.9 (ablation) | Zipper · Normal, p=1000, δ=5, n=500 | empirical power (%): 87.6 -> 100.0 | Helped |
| increase slider τ from 0 to 0.9 (ablation) | Zipper · Binomial, p=5, δ=3, n=500 | empirical power (%): 89.8 -> 100.0 | Helped |
| increase slider τ from 0 to 0.9 (ablation) | Zipper · Binomial, p=1000, δ=5, n=500 | empirical power (%): 78.6 -> 100.0 | Helped |
| increase slider τ to an excessively large value (ablation) | Zipper · Normal and binomial null simulations, n=500, p=1000 | empirical size (%): 5.9 -> 11.1; 5.1 -> 20.9 | Hurt |
| set slider τ=0.99 rather than τ=0 (ablation) | Zipper · Normal null, n=500, p=1000 | empirical size (%): 5.9 -> 11.1 | Hurt |
| set slider τ=0.99 rather than τ=0 (ablation) | Zipper · Binomial null, n=500, p=1000 | empirical size (%): 5.1 -> 20.9 | Hurt |
Surprising findings
- Size inflates for large tau near 1 (e.g., tau=0.99 gives 11.1% size for normal p=1000), showing a tradeoff between power and degeneracy.
- WGSC-2 and DSP-Pert achieve higher power in high dimensions but at the cost of invalid size control.
Limitations
- Under the null, tau close to 1 inflates Type I error due to degeneracy, so tau must be restricted to [0,1).
- The variance estimator is inconsistent under the alternative when 0<tau<1, so confidence intervals require an additional plug-in estimator.
- Selection of tau relies on a heuristic large-sample size n0 (e.g., 30 or 50) rather than a data-adaptive rule.
Experimental setup
Datasets: MNIST
Benchmarks: simulated normal, binomial and t3 models MNIST digits 7 and 9 subset
Metrics: empirical size empirical power
Baselines: WGSC-3 DSP-Split WGSC-2 DSP-Pert
Builds on: LOCO variable importance additional splitting of testing data (DSP-Split) WGSC-3 / WGSC-2 tests data perturbation (DSP-Pert)
Closest papers by meaning
- Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data? (ICLR 2026 · similarity 0.83)
- A Black-Box Debiasing Framework for Conditional Sampling (NeurIPS 2025 · similarity 0.83)
- SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions (NeurIPS 2026 · similarity 0.83)
- Optimal Algorithms for Augmented Testing of Discrete Distributions (NeurIPS 2024 · similarity 0.82)
- Exploiting LLMs for Automatic Hypothesis Assessment via a Logit-Based Calibrated Prior (NeurIPS 2025 · similarity 0.82)
- TEA: Test-time Energy Adaptation (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.