Research map / Large language models / Instruction and fine-tuning
Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pairwise comparison (vs adversarial real/fake scoring) (ablation) | Comparative evaluator · DailyDialog, model-level | Spearman: 0.491 -> 0.764 | Helped |
| add tie option (ablation) | Comparative evaluator · DailyDialog, model-level | Spearman: 0.557 -> 0.764 | Helped |
| add strong supervision examples (ablation) | Comparative evaluator · DailyDialog, model-level | Spearman: 0.697 -> 0.764 | Helped |
| add weak supervision examples (checkpoint ordering) (ablation) | Comparative evaluator · DailyDialog, model-level | Spearman: 0.728 -> 0.764 | Helped |
| add human preference fine-tuning (ablation) | Comparative evaluator · DailyDialog, model-level | Spearman: 0.602 -> 0.764 | Helped |
| initialize from BERT (ablation) | Comparative evaluator · DailyDialog, model-level | Spearman: 0.644 -> 0.764 | Helped |
| use model-level skill rating instead of averaged sample-level skill rating (ablation) | Comparative evaluator · DailyDialog, model-level | Spearman: 0.651 -> 0.764 | Helped |
| use model-level skill rating instead of averaged reference-based score (ablation) | Comparative evaluator · DailyDialog, model-level | Spearman: 0.634 -> 0.764 | Helped |
| use sample-level skill rating vs random reference sampling | Comparative evaluator · DailyDialog, sample-level | Pearson: 0.472 -> 0.469 | No clear effect |
| select hyperparameters with comparative evaluator instead of BLEU | NLG model training · Hyperparameter search | avg human score: 2.75 -> 3.14 | Helped |
| select early-stopping checkpoint with comparative evaluator instead of BLEU | NLG model training · Early-stopping | avg human score: 3.15 -> 3.41 | Helped |
Closest papers by meaning
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.87)
- Generative Judge for Evaluating Alignment (ICLR 2024 · similarity 0.87)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.87)
- Open-Domain Text Evaluation via Contrastive Distribution Methods (ICML 2024 · similarity 0.86)
- Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.86)
- Enabling Lanuguage Models to Implicitly Learn Self-Improvement (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.