Research map / Large language models / Language, safety and interpretability
Open-Domain Text Evaluation via Contrastive Distribution Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Contrastive Distribution Methods (CDM), which contrast expert and amateur language models of different sizes to build reference-free evaluators for open-domain dialogue coherence and commonsense in controllable generation.
Problem
Automatic reference-free evaluation of open-domain text generation remains unreliable; reference-based metrics fail on diverse outputs, and trained discriminators need costly negative samples.
Approach
Use the contrast between a larger expert and smaller amateur model of the same family as an approximation of a quality oracle. Generative CDM degrades the amateur along the contrast direction to synthesize deceptive negatives for training a discriminator; Discriminative CDM pools step-wise contrastive momentum log p_e - log p_a directly as the score.
Key results
- Discriminative CDM (T5 small-11b) reaches Spearman 0.59 coherence / 0.62 overall on FED, versus 0.57/0.54 for the best G-Eval variant (LLaMA2-7b-Vicuna) and 0.47/0.55 for DEAM.
- On CommonGen-trinity raw, Discriminative CDM gets 0.73 Spearman, above G-Eval with Flan-T5-11b (0.61) and Generative CDM (0.63).
- Resampling directly from amateur models yields much weaker metrics (e.g., T5-large 0.09 coherence on FED), showing inverse scaling of naive sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Discriminative CDM rather than Generative CDM | CDM dialogue evaluation · FED and DSTC9 | Spearman correlation: 0.53 -> 0.59; 0.55 -> 0.62; 0.22 -> 0.28; 0.24 -> 0.27 | Helped |
| Use Discriminative CDM rather than Generative CDM | CDM commonsense evaluation · CommonGen-trinity | Spearman correlation: 0.63 -> 0.73; 0.55 -> 0.61 | Helped |
| Add mixed utterance- and segment-level manipulations repeated multiple times (ablation) | Generative CDM · FED coherence | Spearman correlation: 0.32 -> 0.42 | Helped |
| Add AMR-guided manipulation to negative sampling (ablation) | Generative CDM · FED coherence | Spearman correlation: 0.42 -> 0.49 | Helped |
| Use classifier pooling instead of max pooling (ablation) | Discriminative CDM · FED coherence | Spearman correlation: 0.46 -> 0.53 | Helped |
| Use T5-11b rather than T5-base as expert (ablation) | Generative CDM with T5-small amateur · FED coherence | Spearman correlation: 0.48 -> 0.53 | Helped |
| Use T5-11b rather than T5-base as expert (ablation) | Discriminative CDM with T5-small amateur · FED coherence | Spearman correlation: 0.42 -> 0.59 | Helped |
| Replace contrastive negative generation with direct resampling from the amateur model (ablation) | Generative CDM · FED coherence | Spearman correlation: 0.48 -> 0.31 | Hurt |
| Use infilling rather than autoregressive generation for Pythia negative manipulation (ablation) | Generative CDM · DSTC9 coherence | Spearman correlation: 0.09 -> 0.18 | Helped |
Surprising findings
- Sampling negatives directly from larger amateur models produces worse evaluators, an inverse-scaling effect that motivates contrasting against the expert.
- Discriminative CDM with no trained classifier outperforms trained discriminator-based metrics like DEAM and Generative CDM on dialogue benchmarks.
Limitations
- Theoretically cannot evaluate inverse-scaling tasks where bigger models perform worse, since the method assumes a scaling-consistent partial order.
- Relies on a first-order secant-hyperplane approximation of the oracle, which may be limited.
- Requires task-dependent practice to choose amateur/expert pairs and pooling strategy for new domains.
Experimental setup
Datasets: TopicalChat PersonaChat FED DSTC9 CommonGen CommonGen-trinity
Benchmarks: FED DSTC9 CommonGen-trinity
Metrics: Spearman correlation
Baselines: ADEM DEAM DynaEval UniEval G-Eval BOOST ACCENT
Builds on: contrastive decoding DEAM ExPO G-Eval
Closest papers by meaning
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.87)
- RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (ICLR 2025 · similarity 0.86)
- Controlled Text Generation via Language Model Arithmetic (ICLR 2024 · similarity 0.86)
- WritingBench: A Comprehensive Benchmark for Generative Writing (NeurIPS 2025 · similarity 0.85)
- Chunk-Distilled Language Modeling (ICLR 2025 · similarity 0.85)
- Generative Monoculture in Large Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.