Research map / Probabilistic and scientific modeling / Molecules and proteins

Prediction-Powered Inference Across Many Tasks for AI Evaluations and Social Science Research

NeurIPS 2026 · Probabilistic and scientific modeling › Molecules and proteins · approach: Prediction-powered and conformal inference

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes multi-task prediction-powered inference that learns a proxy-to-ground-truth recalibrator from other tasks (GRePPI, ARePPI) while keeping within-task rectification and power tuning, and applies it to auditing LLM demographic steering on 2024 election questions.

Problem

Per-task PPI estimators are inefficient when each task has only a handful of human labels (n≈20–40), since the rectifier variance dominates; naive pooling of rectifiers across tasks biases inference.

Approach

Learn a nonlinear recalibration map (isotonic regression) of proxy scores to outcomes from other tasks (leave-one-task-out for GRePPI; cross-fitted global-local mixture for ARePPI), then apply within-task rectification and power tuning so per-task CIs stay valid; theory shows affine cross-task recalibration gives no gain over PPI++.

Key results

• On semi-synthetic data with n=40 per task, GRePPI and ARePPI yield the tightest Wald CIs and lowest MSE among compared methods, beating RePPI and PPI++ (exact figures in plots, not stated in text).
• Theory: any affine cross-task recalibration has the same oracle variance as the identity proxy; strict gains over power-tuned PPI require the proxy-outcome regression to be nonlinear.
• Power tuning gives poor small-sample coverage at n<=40; fixing lambda=1 improves coverage, so CI-width results on human data use lambda=1.

Surprising findings

• Affine recalibration of the proxy, and hence cross-task linear sharing, gives no asymptotic efficiency gain over power-tuned PPI++.
• Power tuning hurts coverage at very small label sizes, so it is dropped for the human-annotation study.

Limitations

• Small label budgets (n<=40) make power-tuned variance estimates overly optimistic, so coverage suffers unless power tuning is dropped.
• Gains depend on tasks sharing a proxy-ground-truth relationship; heterogeneous tasks can make pooled recalibration worse (ARePPI mitigates this).
• Ground truth is averaged over five annotators and annotation variance is treated as negligible; the 'deep meaning similarity' construct is normative.

Experimental setup

Datasets: 2024 US presidential election LLM responses dataset (Prolific human annotations, 186 questions x 72 tasks) semi-synthetic LLM-judge scores (gpt-5-mini ground truth, gpt-4o-mini surrogate) synthetic power-function tasks f_p(x)=x^p

Benchmarks: semi-synthetic LLM-judge data 2024 US election steering human-annotation study synthetic task-heterogeneity ablation

Metrics: CI width coverage MSE relative MSE vs PPI

Baselines: Classical PPI PPI++ RePPI

Builds on: prediction-powered inference PPI++ RePPI cross-prediction-powered inference

Closest papers by meaning

• Stratified Prediction-Powered Inference for Effective Hybrid Evaluation of Language Models (NeurIPS 2024 · similarity 0.85)
• Inverse Constitutional AI: Compressing Preferences into Principles (ICLR 2025 · similarity 0.85)
• Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.85)
• Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemble (NeurIPS 2026 · similarity 0.85)
• Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regression (ICML 2025 · similarity 0.85)
• Valid Inference with Imperfect Synthetic Data (NeurIPS 2025 · similarity 0.84)