Research map / Large language models / Alignment and preferences
Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Discriminative Fine-Tuning (DFT), which replaces SFT's token-level generative objective with a discriminative softmax over sampled base-model outputs as negatives, optimized via FCCO-style moving-average estimators, without preference data or reward models.
Problem
SFT only increases likelihood of good answers and does not push down likelihood of bad answers, so it is typically followed by preference optimization that needs human preference data or reward models.
Approach
Model the discriminative likelihood of an answer against all possible outputs, approximate the denominator with self-generated outputs from the base LLM (as negatives), and optimize the resulting log-sum-exp objective with a moving-average FCCO estimator; DFT2 further drops the base-model probability weighting for efficiency.
Key results
- DFT achieves 79.15% on GSM8K and DFT2 28.62% on MATH with Mistral-7B, state-of-the-art among 7B models in the paper's comparison, surpassing MetaMath-Mistral-7B (77.7 / 28.2).
- On the seven-benchmark suite, DFT averages 62.84 vs SFT 61.79, and outperforms self-play PO baselines such as SPIN (57.80) and SimPO (51.40) using the same self-generated negatives.
- DFT (single stage, no preference data) is competitive with SFT->DPO/SimPO/KTO under the UF->UF pipeline (e.g., SFT->DPO 62.98, SFT->KTO 63.04 vs DFT 62.84).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace SFT with DFT | Mistral-7B-base · GSM8K | accuracy: 77.7 -> 79.15 | Helped |
| replace SFT with DFT2 | Mistral-7B-base · MATH | accuracy: 28.2 -> 28.62 | Helped |
| use DFT instead of SFT | Mistral-7B-v0.1 · Average across 7 benchmarks | average accuracy: 61.79 -> 62.84 | Helped |
| use DFT instead of self-play preference optimization | Mistral-7B-v0.1 · Average across 7 benchmarks | average accuracy: 57.80 -> 62.84 | Helped |
| use moving-average estimator with gamma in 0.80–0.95 rather than gamma=1 (ablation) | DFT · Average across 7 benchmarks | average accuracy: 52.45 -> 62.84 | Helped |
| set gamma=1 instead of using a moving-average estimator (ablation) | DFT · GSM8K | accuracy: 48.22 -> 22.14 | Hurt |
| increase negative samples from B=1 to B=2 (ablation) | DFT · Average across 7 benchmarks | average accuracy: 62.33 -> 62.84 | Helped |
| increase negative samples from B=2 to B=4 (ablation) | DFT · Average across 7 benchmarks | average accuracy: 62.84 -> 62.47 | Hurt |
| generate negative samples at temperature 0.7 rather than 0 (ablation) | DFT · Average across 7 benchmarks | average accuracy: 62.15 -> 62.84 | Helped |
| use chat-template prompts with an adversarial system message rather than a beneficial system message (ablation) | DFT negative-sample generation · Average across 7 benchmarks | average accuracy: 62.35 -> 62.84 | Helped |
| use DFT rather than SFT | Qwen-2.5-0.5B · Average across 7 benchmarks | average accuracy: 40.63 -> 41.07 | Helped |
| use DFT2 rather than SFT | Llama3-8B-instruct · Average across 7 benchmarks | average accuracy: 67.56 -> 69.42 | Helped |
Surprising findings
- Preference optimization methods applied on self-generated negatives failed to improve over SFT (e.g., SimPO drops GSM8K to 2.58), contrary to prior self-play findings.
- Setting gamma=1 (no moving-average estimator) substantially degrades performance (average 52.45 vs 62.84).
Limitations
- Negatives are sampled from the base model with adversarial prompts, so quality of negatives depends on prompting strategy and temperature.
- DFT2 underperforms DPO and SimPO on AlpacaEval2 win rate, and GPT-4-judged results favor longer outputs; DFT outputs are shorter.
- Gains over SFT on larger or other base models are small (e.g., Qwen-2.5-0.5B +0.44%).
Experimental setup
Datasets: MetaMathQA UltraFeedback UltraChat-200k
Benchmarks: GSM8K MATH MMLU TruthfulQA HellaSwag WinoGrande ARC IFEval AlpacaEval2
Metrics: zero-shot accuracy few-shot accuracy prompt-level strict accuracy length-controlled win rate
Baselines: SFT SPIN SimPO SimPO-SFT KTO ORPO DPO-p DPO RRHF R-DPO CPO IPO
Builds on: SogCLR FCCO DPO SimPO SPIN
Closest papers by meaning
- Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models (AAAI 2025 · similarity 0.93)
- FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain (ICML 2025 · similarity 0.90)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.90)
- Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.90)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.89)
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.