Research map / Probabilistic and scientific modeling / Molecules and proteins
Stratified Prediction-Powered Inference for Effective Hybrid Evaluation of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StratPPI, a stratified extension of prediction-powered inference (PPI++) that yields tighter valid confidence intervals for LLM and other model evaluation by combining small human-labeled samples with autorater labels across strata.
Problem
Estimating the mean performance of language models with limited human labels is costly, and autoraters are biased and heterogeneous across subdomains; standard PPI ignores this heterogeneity, leaving wider confidence intervals than necessary.
Approach
Partition inputs into strata (e.g., by autorater prediction value), apply PPI++ within each stratum with stratum-specific tuning parameters and sample allocations, and combine them via weighted M-estimation; closed-form optimal lambda and sampling rates are derived, and an estimator for allocation is given that needs only autorater confidence scores.
Key results
- On Seahorse, StratPPI Heur. with 300 human ratings matches the confidence of classical inference with 600 human ratings (about 2x effective sample size).
- In simulations with heterogeneous autorater bias or variance across strata, StratPPI substantially outperforms PPI++, which becomes barely more powerful than classical inference.
- Theory shows StratPPI's asymptotic variance (trace) is at most that of PPI++, with equality only when stratum conditional means of Y and f(X) are equal.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Stratify PPI++ by autorater predictions | PPI++ · Seahorse | confidence interval width reduction over classical inference: 0.10 → 0.20 | Helped |
| Stratify PPI++ by autorater predictions | PPI++ · AttributedQA | confidence interval width reduction over classical inference: 0.20 → 0.30 | Helped |
| Stratify PPI++ by autorater predictions | PPI++ · Galaxy | confidence interval width reduction over classical inference: 0.25 → 0.35 | Helped |
| Use heuristic sample allocation proportional to stratum weight times estimated standard deviation | StratPPI · Galaxy | confidence interval width reduction: +10 percent | Helped |
| Use StratPPI Heur. with 300 human ratings instead of classical inference with 600 human ratings | Seahorse · Seahorse | confidence interval width: 300 human ratings; 600 human ratings; 2× | Helped |
| Stratify PPI++ when strata are homogeneous | PPI++ · Synthetic mean estimation, homogeneous strata | confidence interval width | No clear effect |
| Use optimal rather than proportional sample allocation when stratum variances differ | StratPPI · Synthetic mean estimation, heterogeneous variance strata | confidence interval width | Helped |
| Use StratPPI with prediction-based stratification | PPI++ · Synthetic mean estimation, heterogeneous strata | confidence interval width | Helped |
| Use the heuristic allocation with uncalibrated LLM-judge confidence scores | StratPPI Heur. · Chatbot Arena, gpt-4-1106-preview vs. claude-2.1 | confidence interval width | Mixed |
| Use stratification | PPI++ · Chatbot Arena, gpt-4-1106-preview vs. claude-2.1 | confidence interval width | Helped |
Surprising findings
- Heuristic allocation becomes overly aggressive at larger n on Chatbot Arena because LLM-judge confidence scores are poorly calibrated.
Limitations
- Guarantees are asymptotic; finite-sample coverage is not guaranteed.
- Number of strata must be fixed; the sampling model requires fixed-size samples per stratum and differs from i.i.d. settings.
- Benefit disappears if the chosen stratification is not statistically useful, and post-hoc stratification of already collected human data is not covered.
Experimental setup
Datasets: Seahorse AttributedQA Galaxy Chatbot Arena
Benchmarks: Seahorse AttributedQA Galaxy Chatbot Arena
Metrics: confidence interval width coverage percent reduction in CI width effective sample size RMSE
Baselines: classical inference PPI++ active statistical inference
Builds on: prediction-powered inference PPI++ active statistical inference stratified sampling
Closest papers by meaning
- Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regression (ICML 2025 · similarity 0.87)
- Aioli: A Unified Optimization Framework for Language Model Data Mixing (ICLR 2025 · similarity 0.87)
- Anytime-valid, Bayes-assisted, Prediction-Powered Inference (NeurIPS 2025 · similarity 0.86)
- Prediction-Powered Semi-Supervised Learning with Online Power Tuning (NeurIPS 2025 · similarity 0.86)
- Prediction-Powered Inference Across Many Tasks for AI Evaluations and Social Science Research (NeurIPS 2026 · similarity 0.85)
- Valid Inference with Imperfect Synthetic Data (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.