Research map / Large language models / Language, safety and interpretability
Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges
Chen Feng, Minghe Shen, Ananth Balashankar, Carsten Gerner-Beuerle, Miguel R. D. Rodrigues
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use noisy judge-based test instead of direct human-only test with a capable judge | Noisy HT · Jigsaw, Hate Speech, SafeRLHF | type-II error | Improved |
| use noisy test with a weak judge (low TPR / high FPR) | Noisy HT · synthetic and real datasets | type-II error | Hurt |
| add variance-corrected threshold accounting for calibration uncertainty (ablation) | Noisy HT · all settings | type-I error | Improved |
| know judge TPR/FPR exactly (oracle) instead of estimating (ablation) | Noisy HT · synthetic and real datasets | type-II error | Improved |
| use PPI baselines instead of noisy HT | Noisy HT · all settings | type-II error | Hurt |
| increase calibration set size n_M from 25 to 100 (ablation) | TPR/FPR estimators · Jigsaw | estimator standard deviation | Improved |
| use federated prompts instead of single judge prompt (ablation) | LLM judge · Jigsaw | judge TPR/FPR and type-II error | Improved |
Closest papers by meaning
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.91)
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.91)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025 · similarity 0.90)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.90)
- Statistical Hypothesis Testing for Auditing Robustness in Language Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
