Research map / Probabilistic and scientific modeling / Molecules and proteins
Handling Missing Responses under Cluster Dependence with Applications to Language Model Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes the doubly robust estimator for missing responses under cluster dependence, proving asymptotic normality with diverging cluster sizes and a sequential-MAR extension using history summaries, validated on simulations and OpenAssistant conversation trees.
Problem
Estimating average human annotation scores (e.g., for LLM evaluation) when outcomes are missing and observations are clustered (e.g., multi-turn conversations per user or conversation tree), so standard i.i.d. inference is invalid.
Approach
Extends doubly robust estimation to cluster-dependent data with unbounded/diverging cluster sizes, showing the convergence rate sqrt(n/Omega_n) adapts to within-cluster dependence (faster than sqrt(G) when dependence is weak), and proposes a sequential MAR assumption using history summaries S_gt for missingness and outcome models.
Key results
- In simulations, Wald CIs using the cluster-robust variance estimator attain near-nominal 95% coverage, while i.i.d. variance estimates undercover as within-cluster dependence grows.
- On oasst1 under homogeneous sampling, the i.i.d. doubly robust CI for humor fails to cover the ground-truth average, while the cluster-robust CI covers it.
- Under sequential missingness, cluster-aware CIs are at least 10% wider than i.i.d.-based CIs on oasst1; the history-adjusted DR estimator has lower RMSE than one adjusting only for current information.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Keep μ correctly specified while misspecifying π in the DR estimator (ablation) | DR · Simulation (n=10000, n_g=100) | MSE: 0.0563 -> 0.0563 | No clear effect |
| Keep π correctly specified while misspecifying μ in the DR estimator (ablation) | DR · Simulation (n=10000, n_g=100) | MSE: 0.0563 -> 0.0572 | Hurt |
| Misspecify both μ and π in the DR estimator (ablation) | DR · Simulation (n=10000, n_g=100) | MSE: 0.0563 -> 2.5765 | Hurt |
| Keep μ correctly specified while misspecifying π in the DR estimator (ablation) | DR · Simulation (n=10000, n_g=10) | MSE: 0.0230 -> 0.0230 | No clear effect |
| Keep π correctly specified while misspecifying μ in the DR estimator (ablation) | DR · Simulation (n=10000, n_g=10) | MSE: 0.0230 -> 0.0233 | No clear effect |
| Misspecify both μ and π in the DR estimator (ablation) | DR · Simulation (n=10000, n_g=10) | MSE: 0.0230 -> 2.3102 | Hurt |
| Keep μ correctly specified while misspecifying π in the DR estimator (ablation) | DR · Simulation (n=1000, n_g=31) | MSE: 0.3248 -> 0.3246 | No clear effect |
| Keep π correctly specified while misspecifying μ in the DR estimator (ablation) | DR · Simulation (n=1000, n_g=31) | MSE: 0.3248 -> 0.3470 | Hurt |
| Misspecify both μ and π in the DR estimator (ablation) | DR · Simulation (n=1000, n_g=31) | MSE: 0.3248 -> 5.3977 | Hurt |
Surprising findings
- The DR estimator can converge faster than the sqrt(G) rate when within-cluster dependence is weak and cluster sizes are large.
Limitations
- DR estimator can be unstable when propensity scores are near zero
- Worst-case variance bound in corollaries may be conservative when independence holds within clusters
- Real-data missingness is simulated via a logistic model rather than observed, so the missingness mechanism is synthetic
Experimental setup
Datasets: OpenAssistant Conversations (oasst1)
Benchmarks: OpenAssistant Conversations (oasst1)
Metrics: coverage probability of 95% confidence intervals RMSE MSE confidence interval width
Baselines: unadjusted complete-case estimator i.i.d. doubly robust variance regression (outcome model) estimator inverse propensity weighting
Builds on: doubly robust estimation central limit theorem for clustered data sensitivity analysis for omitted variable bias
Closest papers by meaning
- Valid Inference with Imperfect Synthetic Data (NeurIPS 2025 · similarity 0.86)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.85)
- Human Feedback is not Gold Standard (ICLR 2024 · similarity 0.85)
- Mind the Uncertainty in Human Disagreement: Evaluating Discrepancies Between Model Predictions and Human Responses in VQA (AAAI 2025 · similarity 0.85)
- Mitigating Social Bias in Large Language Models: A Multi-Objective Approach Within a Multi-Agent Framework (AAAI 2025 · similarity 0.85)
- Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.