Research map / Large language models / Alignment and preferences
Predicting Human-Gain Curves under Local Proxy Optimization from Repeated Ratings
Moonwon Choi, Kisung Nam, Seunggeun Lee
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a pre-optimization diagnostic that uses repeated human ratings to predict how human-perceived quality changes as a fixed proxy objective is optimized more strongly.
Problem
Average proxy-human correlation does not reveal how human-perceived quality changes as optimization strength increases under a proxy objective.
Approach
Models proxy optimization locally and uses repeated human ratings to predict the human-gain curve induced by a fixed proxy before optimization is performed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add curvature term B to A-only forecast (ablation) | PACE · SummEval human | endpoint MAE at d=0.10: 0.0310 -> 0.0174 | Improved |
| add curvature term B to A-only forecast (ablation) | PACE · WMT MQM | endpoint MAE at d=0.10: 0.1298 -> 0.0846 | Improved |
| add curvature term B to A-only forecast (ablation) | PACE · OpenMEVA-MAGS | pre-pool CurveMAE: 0.01058 -> 0.00991 | No clear effect |
| add curvature term B to A-only forecast (ablation) | PACE · SummEval, GPT-4o, WMT, OpenMEVA | CurveMAE reduction: 34.3%, 21.7%, 26.4%, 6.4% | Improved |
| use proxy-human alignment instead of reliability-only A_nu (ablation) | PACE · WMT MQM | endpoint MAE at d=0.10: 0.6230 -> 0.0846 | Improved |
| replace PACE with binned calibration (ablation) | Flexible calibration · SummEval human | pre-pool CurveMAE: 0.01185 -> 0.03010 | Hurt |
| replace PACE with isotonic regression (ablation) | Flexible calibration · WMT MQM | pre-pool CurveMAE: 0.05921 -> 0.05882 | No clear effect |
| provide target-pool proxy scores to isotonic calibrator | Isotonic calibration · SummEval human | target-pool-aware CurveMAE: 0.01185 -> 0.00909 | Improved |
| use projection instead of direct integration (ablation) | Calibrated curve · 12 dataset-calibrator cells | CurveMAE: 0.00002-0.00286 difference | No clear effect |
| add target calibration to source-only coefficients | PACE transfer · SummEval to WMT | endpoint MAE: 0.0436 -> 0.0259 | Improved |
| add second expert rating to calibration | PACE B estimate · SummEval | mean absolute error of B-hat: 0.0618 -> 0.0428 | Improved |
| 1% positive 8-SD proxy contamination | PACE B estimate · OpenMEVA | correlation with clean B: -0.11 | Hurt |
Closest papers by meaning
- Position: Human Baselines in Model Evaluations Need Rigor and Transparency (With Recommendations & Reporting Checklist) (ICML 2025 · similarity 0.85)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.85)
- Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization (NeurIPS 2026 · similarity 0.84)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.84)
- Conformal Prediction Sets Improve Human Decision Making (ICML 2024 · similarity 0.84)
- Model Alignment as Prospect Theoretic Optimization (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
