Research map / Probabilistic and scientific modeling / Molecules and proteins
Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows PPI++ can underperform classical inference with few labels, reinterprets its lambda as post-hoc regression, and proposes Ridge-PPI and Sigmoid-PPI estimators that reduce variance in the low-label regime.
Problem
Prediction-powered inference (PPI++) can have higher error than classical sample-mean estimation when few labelled examples are available, because the estimated tuning parameter lambda has high variance.
Approach
Reinterpret PPI's lambda as a post-hoc regression coefficient and replace OLS with regularized (ridge) or nonlinear (sigmoidal) post-hoc regressors to reduce the variance of the lambda estimate and thus of the PPI mean estimate.
Key results
- On the LLM refusal data, Ridge-PPI and Sigmoid-PPI improve over PPI++ for all small labelled sizes and can reduce MAE of classical estimation by over a quarter at small n.
- On distributions where PPI++ is worse than classical estimation (up to 20% more error), Ridge-PPI and Sigmoid-PPI perform about 10% better than classical estimation.
- Across research datasets, Ridge-PPI and Sigmoid-PPI match or exceed both classical estimation and PPI++; correlation between MSE ratio and Var[f(X)] is Pearson r = -0.69.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Ridge-PPI instead of PPI++ | PPI++ · Research datasets (plankton, alphafold, forest, ballots, census_healthcare) | MAE | Helped |
| use Sigmoid-PPI instead of PPI++ | PPI++ · Research datasets (plankton, alphafold, forest, ballots, census_healthcare) | MAE | Helped |
| use PPI++ instead of classical estimation | Classical estimation · LLM Refusal Dataset, small labelled sets | MAE | Helped |
| use Ridge-PPI instead of PPI++ | PPI++ · LLM Refusal Dataset, small labelled sets | MAE | Helped |
| use Sigmoid-PPI instead of PPI++ | PPI++ · LLM Refusal Dataset, small labelled sets | MAE | Helped |
| use Ridge-PPI instead of classical estimation | Classical estimation · LLM Refusal Dataset, small labelled sets | MAE: over a quarter | Helped |
| use Sigmoid-PPI instead of classical estimation | Classical estimation · LLM Refusal Dataset, small labelled sets | MAE: over a quarter | Helped |
| use PPI++ instead of classical estimation | Classical estimation · LLM refusal distributions where PPI++ succeeds | MAE: roughly 30% | Helped |
| use PPI++ instead of classical estimation | Classical estimation · LLM refusal distributions where PPI++ fails | MAE: 20% more error | Hurt |
| use Ridge-PPI or Sigmoid-PPI instead of classical estimation | Classical estimation · LLM refusal distributions where PPI++ fails | MAE: 10% better | Helped |
| increase the labelled batch size | Ridge-PPI · LLM Refusal Dataset, larger labelled batches | MAE: n=20 | No clear effect |
| increase the labelled batch size | Sigmoid-PPI · LLM Refusal Dataset, large labelled batches approaching the unlabelled dataset size | MAE | Hurt |
Surprising findings
- PPI++ can perform worse than classical inference when labelled data is scarce.
- The variance of PPI++ scales inversely with Var[f(X)] under their analysis, counter to intuition.
Limitations
- Sigmoid-PPI performance decays as labelled set size approaches unlabelled set size, with potential asymptotic bias; asymptotic guarantees do not directly apply to Sigmoid-PPI.
- Theoretical analysis relies on simplifying independence and convergence assumptions; the variance expression is non-convex in alpha.
- Correlation with Var[f(X)] across datasets is not statistically significant given few datapoints.
Experimental setup
Datasets: LLM Refusal Dataset (over 50,000 prompt-answer pairs with human refusal annotations) Research datasets aggregated by prior PPI work (plankton, alphafold, forest, ballots, census_healthcare, galaxies)
Benchmarks: LLM Refusal Dataset Research datasets (plankton, alphafold, forest, ballots, census_healthcare)
Metrics: mean absolute error standard deviation of estimates MSE ratio
Baselines: Classical estimation PPI++
Builds on: Prediction Powered Inference PPI++ ridge regression cross-fitting
Closest papers by meaning
- Prediction-Powered Semi-Supervised Learning with Online Power Tuning (NeurIPS 2025 · similarity 0.91)
- Anytime-valid, Bayes-assisted, Prediction-Powered Inference (NeurIPS 2025 · similarity 0.89)
- Prediction-Powered Adaptive Shrinkage Estimation (ICML 2025 · similarity 0.88)
- Stratified Prediction-Powered Inference for Effective Hybrid Evaluation of Language Models (NeurIPS 2024 · similarity 0.87)
- Synthetic-powered predictive inference (NeurIPS 2025 · similarity 0.86)
- Prediction-Powered Active Testing (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.