Research map / Learning theory and algorithms / Prediction and decision losses
AutoEval Done Right: Using Synthetic Data for Model Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies prediction-powered inference (PPI and PPI++) to autoevaluation, combining a small human-labeled set with a large AI-labeled set to get unbiased, lower-variance estimates of model metrics and Bradley-Terry rankings.
Problem
Evaluating ML models with human-labeled validation data is expensive; AI-labeled synthetic data can reduce annotation needs but is biased, so naive use gives invalid estimates and confidence intervals.
Approach
Uses prediction-powered inference with a tunable discount lambda on synthetic labels to debias the synthetic-data estimator, with PPI++ optimizing lambda for variance; extends to pairwise comparisons via Bradley-Terry logistic regression and to distributional outputs of annotator models.
Key results
- On ImageNet with ResNets, PPI and PPI++ improved MSE over the classical estimator and gave about 50% effective sample size gain.
- On protein fitness ranking, PPI++ gave roughly 50% higher ESS than classical and a five-fold rank improvement at n=1000.
- On Chatbot Arena with gpt-4o-mini as judge, PPI++ showed a 20% to 25% ESS improvement over classical Bradley-Terry estimation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use synthetic model scores with bias correction (PPI++) instead of the classical estimator | ImageNet accuracy estimation · ImageNet validation | mean-squared error | Helped |
| Optimize λ for variance rather than fixing λ = 1 (ablation) | PPI++ versus PPI · ImageNet validation | mean-squared error | No clear effect |
| Use synthetic labels for accuracy estimation | PPI and PPI++ · ImageNet validation | effective sample size: approximately 50% | Helped |
| Use optimized λ in PPI++ rather than the classical estimator | Protein fitness correlation estimation · ProteinGym SPG1_STRSG_Olson_2014 | effective sample size: approximately 50% | Helped |
| Use PPI instead of the classical estimator (ablation) | Protein fitness correlation estimation · ProteinGym SPG1_STRSG_Olson_2014 | effective sample size | Hurt |
| Use a better synthetic annotator model | PPI++ protein fitness estimation · ProteinGym SPG1_STRSG_Olson_2014 | effective sample size | Helped |
| Use CARP as the synthetic annotator | PPI++ protein fitness estimation · ProteinGym SPG1_STRSG_Olson_2014 | effective sample size: 10% | Helped |
| Use PPI++ rather than the classical estimator | LLM Bradley-Terry coefficient estimation · Chatbot Arena pairwise preferences | effective sample size: 20% to 25% | Helped |
| Use alternate LLM judges to produce synthetic preferences | PPI++ · Chatbot Arena pairwise preferences | effective sample size: 20% to 35% | Helped |
| Reweight labeled examples for covariate shift (ablation) | AutoEval · ImageNet under covariate shift | coverage: 0.5044 -> 0.9128; 0.3780 -> 0.9196; 0.2444 -> 0.9184; 0.1992 -> 0.9376; 0.1692 -> 0.9252; 0.1572 -> 0.9320 | Helped |
| Use fixed λ = 1 rather than optimizing λ (ablation) | PPI for ImageNet · ImageNet, n=10,000 | efficiency ratio: 1.00 -> 1.27 | Helped |
| Use optimized λ (PPI++) rather than fixed λ = 1 (PPI) (ablation) | ImageNet accuracy estimation · ImageNet, n=10,000 | MSE (× 10^-5): 1.07 -> 1.03 | Helped |
Surprising findings
- The original PPI estimator performed worse than the classical approach on the protein fitness task, which PPI++ mitigates.
- Under covariate shift, unweighted AutoEval coverage collapsed (e.g., 0.16 at n=500 for nominal 0.9), while reweighting restored near-nominal coverage.
Limitations
- Requires labeled and unlabeled data to be representative of production; covariate shift invalidates confidence intervals unless reweighted with known density ratio.
- Synthetic label quality strongly affects gains; with poor annotators the method falls back to classical estimation.
- Asymptotic confidence intervals only, valid as n and N grow large.
Experimental setup
Datasets: ImageNet validation set ProteinGym SPG1_STRSG_Olson_2014 assay Chatbot Arena dataset (16K observations)
Benchmarks: ImageNet validation accuracy ProteinGym protein G fitness Chatbot Arena pairwise preferences
Metrics: MSE effective sample size confidence interval coverage confidence interval width rank correlation
Baselines: classical estimator PPI rank-sets
Builds on: prediction-powered inference PPI++ Bradley-Terry model
Closest papers by meaning
- Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees (NeurIPS 2025 · similarity 0.88)
- Energy-based Automated Model Evaluation (ICLR 2024 · similarity 0.85)
- Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks (ICLR 2025 · similarity 0.84)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.83)
- Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory (ICLR 2025 · similarity 0.83)
- Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.