Research map / Large language models / Language, safety and interpretability
Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas
Luke Guerdan, Justin Whitehouse, Kimberly Truong, Ken Holstein, Steven Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Riesz-loss direct estimation of the reweighting term instead of separate density-ratio and propensity models (ablation) | doubly-robust estimator · PRISM and DICES (text embeddings) | coverage and bias (MAE) | Improved |
| use doubly-robust estimator rather than PPI++/RePPI/Source Mean under evaluation sampling bias | system quality estimation · Synthetic, PRISM, DICES | coverage of 95% CIs | Improved |
| use persona ratings with higher correlation to human ratings (ablation) | DR (Riesz) · DICES (real LLM personas) | coverage: rho 0.23 -> 0.43 (GPT-4o Mini -> GPT-5) | Improved |
| use DR (Riesz) rather than RePPI with real LLM personas | persona-based estimation · DICES (real LLM personas) | coverage | Improved |
| increase dropout rate in source data (ablation) | baseline estimators · Synthetic | coverage | Mixed |
Closest papers by meaning
- Validating LLM-as-a-Judge Systems under Rating Indeterminacy (NeurIPS 2025 · similarity 0.89)
- Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts (NeurIPS 2026 · similarity 0.86)
- Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions (AAAI 2025 · similarity 0.86)
- LLM Generated Persona is a Promise with a Catch (NeurIPS 2025 · similarity 0.86)
- Feedback Forensics: A Toolkit to Measure AI Personality (NeurIPS 2026 · similarity 0.86)
- Diverse Yet Biased: Towards Mitigating Biases in Generative AI (Student Abstract) (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
