Research map / Label-efficient and robust learning / Recognition and long-tail
Better than Random: Reliable NLG Human Evaluation with Constrained Active Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CASF, an active, constrained systematic sampling framework that selects representative subsets for NLG human evaluation so the resulting system ranking better matches the full-dataset ranking.
Problem
Human evaluation on small random subsets of NLG data yields inter-system rankings that vary with the subset chosen, making the reported system ranking unreliable.
Approach
A Constrained Active Sampling Framework combining a GBDT Learner that predicts sample quality from automatic metric scores, a Systematic Sampler that buckets samples by predicted quality, and a Constrained Controller that picks one non-redundant sample per bucket (bi-gram redundancy), run over multiple phases with human labels fed back to the Learner.
Key results
- CASF achieves 93.18% top-ranked system recognition accuracy across 44 human metrics and 137 systems.
- CASF ranks first or second on 90.91% of human metrics, with overall inter-system Kendall correlation of 0.83 versus 0.68 for random sampling.
- Phases experiment: 5 phases with 10% per-phase sampling ratio gives the best Kendall tau (0.83) in the average mode.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CASF instead of random sampling | NLG human-evaluation subset selection · 16 datasets, 44 human metrics | Kendall's Tau: 0.68 -> 0.83 | Helped |
| use CASF instead of heuristic sampling | NLG human-evaluation subset selection · 16 datasets, 44 human metrics | Kendall's Tau: 0.72 -> 0.83 | Helped |
| remove the Constrained Controller (ablation) | CASF · 5 NLG tasks, 44 human metrics | top-ranked system accuracy: 0.93 -> 0.77 | Hurt |
| remove the Constrained Controller (ablation) | CASF · 16 datasets, 44 human metrics | Kendall's Tau: 0.83 -> 0.68 | Hurt |
| use only the preliminary phase with eight automatic metrics (ablation) | CASF · 5 NLG tasks, 44 human metrics | top-ranked system accuracy: 0.93 -> 0.84 | Hurt |
| use only the preliminary phase with one automatic metric (ablation) | CASF · 5 NLG tasks, 44 human metrics | top-ranked system accuracy: 0.93 -> 0.91 | Hurt |
| replace GBDT Learner with linear regression (ablation) | CASF Learner · 44 human metrics, 5 NLG tasks | Kendall's Tau: 0.828 -> 0.833 | Hurt |
| use MOVER-SCORE instead of METEOR for preliminary-phase quality ranking (ablation) | CASF · CASF preliminary-phase pre-ranking, all tasks | Kendall's Tau: 0.79 -> 0.83 | Helped |
| use 5 phases instead of 4 in average mode (ablation) | CASF · CASF, 44 human metrics | Kendall's Tau: 0.76 -> 0.83 | Helped |
| use 5 phases instead of 6 in average mode (ablation) | CASF · CASF, 44 human metrics | Kendall's Tau: 0.72 -> 0.83 | Helped |
Surprising findings
- Random sampling yields different inter-system rankings across repeated draws for 87.5% of datasets.
- Only 60.7% of surveyed papers that sampled for human evaluation used random sampling; 39.3% did not state their sampling method.
Limitations
- Relies on automatic metric scores as features for quality prediction, which may not transfer to tasks with weak metrics.
- Evaluated only on Likert-scale human evaluation setups; other comparison types are not covered.
- Redundancy measured by bi-gram similarity, a simple proxy.
Experimental setup
Datasets: SummEval REALSumm Newsroom (NeR18) DialSummEval OpenAI summarization axis datasets WMT newstest2020/2021 Persona Chat human eval MANS-ROC MANS-WP THUMB-MSCOCO VATEX-EVAL E2E
Benchmarks: SummEval REALSumm Newsroom (NeR18) DialSummEval OpenAI summarization axis datasets WMT newstest2020/2021 Persona Chat human eval MANS-ROC MANS-WP THUMB-MSCOCO VATEX-EVAL
Metrics: Kendall's Tau top-ranked system accuracy Wilcoxon signed-rank test
Baselines: Random Sampling Heuristic Sampling (ILDAE) Eight Metric Single Metric Online Sampling
Builds on: systematic sampling active evaluation MoverScore
Closest papers by meaning
- Elo Uncovered: Robustness and Best Practices in Language Model Evaluation (NeurIPS 2024 · similarity 0.85)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.85)
- Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.85)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.84)
- Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (ICLR 2025 · similarity 0.84)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.