Research map / Probabilistic and scientific modeling / Molecules and proteins
Valid Inference with Imperfect Synthetic Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a GMM-based estimator that combines real labeled data with LLM-generated proxy and synthetic samples for valid statistical inference, showing MSE gains when synthetic residuals predict real residuals.
Problem
Combining imperfect LLM-generated synthetic samples with a small human-labeled dataset for statistically valid inference (consistency and proper coverage) in computational social science settings.
Approach
Augment the GMM moment conditions with auxiliary parameters for proxy and synthetic data, where synthetic samples are generated conditioned on individual real texts so their moment residuals correlate with real residuals; the optimal two-step GMM weighting then exploits these cross-moment covariances without a tuning hyperparameter.
Key results
- GMM-Synth achieves the lowest MSE on 8 of 8 downstream tasks, with reductions of more than 50% in MSE in low-label regimes while maintaining valid coverage.
- GMM-Synth improves over GMM-Proxy across all considered tasks, showing synthetic data adds benefit beyond proxy labels.
- PPI++Synth over PPI++Proxy yields much smaller gains, with no MSE gains in the remaining 5 tasks; effective sample savings up to more than 50%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add synthetic-data moments to GMM-Proxy | GMM-Proxy · 8 downstream tasks | MSE: 8 out of 8 | Helped |
| Add synthetic-data moments to GMM-Proxy | GMM-Proxy · 8 downstream tasks | confidence interval width: 7 out of 8 | Helped |
| Add synthetic-data moments to GMM-Proxy | GMM-Proxy · 8 downstream tasks | coverage | No clear effect |
| Add synthetic data using PPI++Synth rather than PPI++Proxy | PPI++Proxy · 5 of 8 downstream tasks | MSE: 5 tasks | No clear effect |
| Add synthetic data using PPI++Synth rather than PPI++Proxy | PPI++Proxy · 3 of 8 downstream tasks | MSE: 3 tasks | Mixed |
| Use proxy data and synthetic data alone without the real-data correction | proxy- and synthetic-data estimators · All settings | estimation bias | Hurt |
| Use synthetic data whose residuals are independent of real-data residuals | GMM-Synth · Asymptotic inference | variance | No clear effect |
| Use synthetic-data residuals predictive of real-data residuals | GMM-Synth · Asymptotic inference | variance | Helped |
Surprising findings
- Adding moments from purely synthetic data, defined with a separate parameter, can improve the target parameter estimate even though it does not change the first-step estimate.
- Proxy-only and synthetic-only estimates are greatly biased, yet combining them with real data via GMM still reduces variance.
Limitations
- Performance depends on the quality of the generative model; very poor synthetic data yields little or no efficiency gain.
- Guarantees are asymptotic and may undercover in extremely low-data regimes.
- RePPI and PPI++ adaptations required ad hoc choices (linear regression or XGBoost for the score model, cross-fitting struggles with small data).
Experimental setup
Datasets: Stack Exchange and Wikipedia politeness requests climate-related news headlines stance corpus congressional bills texts with DW-NOMINATE scores
Benchmarks: Stack Exchange/Wikipedia politeness (hedging, first-person plural) climate stance headlines congressional bills
Metrics: mean-squared error coverage confidence interval width effective sample size
Baselines: PPI++Proxy PPI++Synth PPI++Synth (Oracle) RePPI GMM-Proxy
Builds on: prediction-powered inference PPI++ RePPI generalized method of moments
Closest papers by meaning
- Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory (ICLR 2025 · similarity 0.87)
- Debiasing Synthetic Data Generated by Deep Generative Models (NeurIPS 2024 · similarity 0.87)
- Unlocking Post-hoc Dataset Inference with Synthetic Data (ICML 2025 · similarity 0.86)
- Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective (ICLR 2025 · similarity 0.86)
- Large Language Bayes (NeurIPS 2025 · similarity 0.86)
- Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.