Research map / Large language models / AI and society
AtC: Aggregate-then-Calibrate for Human-centered Assessment
Zejun Xie, Xintong Li, Guang Wang, Desheng Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| calibrate model scores onto the human consensus ranking (AtC) vs model-only | AtC with HRA-E · Reading-level (semi-synthetic) | MSE: 29.00 -> 8.191 | Improved |
| calibrate model scores onto the consensus ranking vs model-only | AtC with HRA-G · Reading-level (semi-synthetic) | Wasserstein: 2.831 -> 0.839 | Improved |
| calibrate model scores onto consensus ranking vs human-only | AtC with HRA-E · Reading-level (semi-synthetic) | Kendall tau: 0.375 -> 0.403 | Improved |
| use heterogeneous Stage-1 (HRA-G) instead of homogeneous BTL (ablation) | AtC · Reading-level (semi-synthetic) | Kendall tau: 0.373 -> 0.410 | Improved |
| use heterogeneous Stage-1 (HRA-G) instead of homogeneous TCV (ablation) | AtC · Reading-level (semi-synthetic) | Wasserstein: 0.914 -> 0.839 | Improved |
| heterogeneous HRA human-only scores versus calibrated, on distance metric | human-only s* · Reading-level (semi-synthetic) | Wasserstein: 2.250 vs 0.839 | Hurt |
| calibrate onto consensus ranking vs human-only | AtC with HRA-E · Dots-activity | Kendall tau: 0.922 -> 0.943 | Improved |
| calibrate onto consensus ranking vs model-only | AtC with HRA-E · Dots-activity | MSE: 11.75 -> 9.59 | Improved |
| AtC vs GPPL | AtC with HRA-E · Dots-activity | MSE: 4220.36 -> 9.59 | Improved |
| add 400 or fewer pairwise inversions to consensus (ablation) | AtC · Reading-level (semi-synthetic) | Kendall tau: decreases steadily | Hurt |
| add more than 500 pairwise inversions to consensus (ablation) | AtC · Reading-level (semi-synthetic) | Kendall tau: becomes negative | Hurt |
| calibrate with aggregated ratings instead of rankings (ablation) | AtC · Dots-activity | Kendall tau | Hurt |
Closest papers by meaning
- Position: Human Baselines in Model Evaluations Need Rigor and Transparency (With Recommendations & Reporting Checklist) (ICML 2025 · similarity 0.86)
- Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge (ICLR 2025 · similarity 0.84)
- BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments (NeurIPS 2026 · similarity 0.83)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.83)
- Can We Trust Item Response Theory for AI Evaluation? (NeurIPS 2026 · similarity 0.83)
- MetaMetrics: Calibrating Metrics for Generation Tasks Using Human Preferences (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
