Research map / Large language models / Instruction and fine-tuning
Benchmarking and Improving Generator-Validator Consistency of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines generator-validator consistency (GV-consistency) for LMs, shows even GPT-4 is inconsistent often, and proposes consistency fine-tuning on filtered consistent generator-validator pairs, which raises Alpaca-30B consistency from 60% to ~94% and improves generator and validator accuracy without labels.
Problem
Language models often generate an answer correctly but then judge that same answer incorrect when validated, which erodes trust and undermines reward models and classifiers built on validator queries.
Approach
Measure consistency between a generator query and its paired validator query, using randomized correctness or option order to avoid trivial always-yes validators, then finetune the LM only on generator-validator response pairs that agree, iterating over rounds, with no labeled data.
Key results
- Consistency fine-tuning raises Alpaca-30B average GV-consistency from 65.1% to 94.1% after three iterations across 6 tasks.
- GPT-4 achieves only 75.8% average GV-consistency, and Alpaca-30B 59.9% in the initial evaluation.
- Consistency fine-tuning improves generator quality by about 14% and validator accuracy by about 6.5% on average without labeled data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Filter fine-tuning pairs to keep only GV-consistent generator and validator responses | Alpaca-30B · Average across 6 tasks | GV-consistency: 65.1 -> 94.1 | Helped |
| Fine-tune on unfiltered generated pairs rather than filtering for consistency | Alpaca-30B · Average across 6 tasks | GV-consistency: 65.1 -> 63.4 | No clear effect |
| Continue consistency fine-tuning to iteration 2 (ablation) | Alpaca-30B consistency fine-tuning · Average across 6 tasks | GV-consistency: 85.9 -> 93.0 | Helped |
| Continue consistency fine-tuning to iteration 3 (ablation) | Alpaca-30B consistency fine-tuning · Average across 6 tasks | GV-consistency: 93.0 -> 94.1 | Helped |
| Condition training sequences on the consistency label instead of filtering to consistent pairs (ablation) | Alpaca-30B · Average across 6 tasks | GV-consistency: 85.9 -> 69.1 | Hurt |
| Apply consistency fine-tuning to Alpaca-7B (ablation) | Alpaca-7B · Style Transfer and Harmful Questions | GV-consistency | Helped |
| Apply consistency fine-tuning to Alpaca-7B (ablation) | Alpaca-7B · HarmfulQ validator | validator accuracy: 5% | Hurt |
Surprising findings
- Even GPT-4 is GV-consistent only about 76% of the time, and some novel tasks sit near the 50% chance level.
- Class-conditioned fine-tuning underperforms filtering-based consistency fine-tuning, and self-training on unfiltered data does not improve consistency.
Limitations
- Validator responses are restricted to binary labels, so feedback is coarse.
- Consistency fine-tuning gives smaller or no generator gains on smaller models such as Alpaca-7B, and sometimes underperforms self-training on validator accuracy.
- Consistency can hold even when both generator and validator are wrong, so consistency does not guarantee accuracy.
Experimental setup
Datasets: TriviaQA Natural Questions synthetic arithmetic problems generated harmful questions generated persona-task prompts generated style transfer inputs
Benchmarks: arithmetic plan arithmetic prompt prioritization QA (TriviaQA) style transfer harmful questions Natural Questions (extrapolation)
Metrics: GV-consistency generator quality validator accuracy exact match
Baselines: Alpaca-30B SelfTrain CC-FT GPT-4 GPT-3.5 text-davinci-003 Alpaca-7B
Builds on: self-training co-training Constitutional AI LoRA
Closest papers by meaning
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.88)
- Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models (ICLR 2024 · similarity 0.85)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.85)
- Reliable and Efficient Amortized Model-based Evaluation (ICML 2025 · similarity 0.85)
- Logical Consistency of Large Language Models in Fact-Checking (ICLR 2025 · similarity 0.85)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.