Research map / Large language models / Alignment and preferences

As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss

ICLR 2025 · Large language models › Alignment and preferences · approach: DPO design choices · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Bidirectional Negative Feedback (BNF), a hyper-parameter-free, pairwise-free alignment loss implemented as a length-normalized cross-entropy with a dynamic target distribution, which reduces alignment tax on reasoning benchmarks.

Problem

DPO-style preference optimization is unstable and hyper-parameter sensitive, especially on math preference data with near-identical pairs, and causes reasoning degradation (alignment tax) when it overfits preferences.

Approach

Replaces pairwise contrastive losses with a single-sample cross-entropy whose target distribution is dynamic (computed from the policy/reference ratio with stop-gradient), so the gradient magnitude decreases in both directions of likelihood change, giving bidirectional negative feedback; works with labeled single responses and needs no extra hyper-parameters.

Key results

• BNF on Gemma-2-9B-it reaches Wild-Bench Elo 1186 and Arena-Hard LC 77.5%, ranking first on Elo and LC among its compared methods, and its average rank is 1.50 (best) across Gemma-2 metrics.
• Across three base models, BNF achieves the best average rank over all metrics (2.88 on Llama-3, 3.00 on Mistral, 1.50 on Gemma-2), while SimPO drops GSM8K accuracy on Llama-3 from 78.5 to 56.6.
• Without any pairwise data (0% pairing ratio), BNF still improves Wild-Bench score by about 6.8 and Arena-Hard win rate by 12.3 points over the Llama-3 base model, while GSM8K rises 1.6 points.

Measured effects reported in the paper

Surprising findings

• Even with 0% pairing ratio (no pairwise data), BNF improves QA benchmarks over the base model and also raises math reasoning scores, reversing the usual alignment tax pattern.
• BNF shows large logit shifts with minimal log-likelihood shift for many samples, attributed to uniform logit increases across preferred tokens.

Limitations

• Only evaluated on 7B-9B models; scalability to 30B+ is unknown.
• Non-pairwise setting is simulated by randomly masking responses from existing pairs rather than real unpaired data.
• Not combined with pairwise contrastive terms, which the authors suggest could be complementary.

Experimental setup

Datasets: UltraFeedback Mistral-Ultrafeedback-PairRM Llama3-Ultrafeedback-ArmoRM Gemma2-Ultrafeedback-ArmoRM

Benchmarks: Wild-Bench Arena-Hard GSM8K MATH-L5 CRUX MMLU-redux

Metrics: Elo Wild-Bench score length-controlled win rate win rate accuracy average rank log-likelihood shift logit shift Gini coefficient

Baselines: SLiC-HF DPO IPO KTO ORPO SimPO RDPO

Builds on: DPO SimPO SLiC-HF KTO IPO

Closest papers by meaning

• Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
• $f$-GRPO & Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment (NeurIPS 2026 · similarity 0.91)
• Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.90)
• On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization (NeurIPS 2025 · similarity 0.90)
• Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · similarity 0.90)
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (ICML 2024 · similarity 0.90)