Research map / Large language models / Alignment and preferences
Noise Contrastive Alignment of Language Models with Explicit Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfoNCA and NCA, two noise-contrastive alignment losses that learn LM policies from explicit scalar reward datasets (and pairwise preferences), showing DPO is a special case of InfoNCA and NCA avoids the decreasing chosen-likelihood problem.
Problem
Existing preference alignment methods such as DPO only handle pairwise preference data, and reward datasets with multiple scalar-rated responses must be pruned into pairs, discarding information; DPO-style training also tends to decrease the likelihood of preferred responses.
Approach
Derives alignment losses from InfoNCE (InfoNCA, a multi-class softmax over K responses weighted by reward softmax) and from NCE (NCA, a per-response binary classification with absolute likelihood targets); DPO emerges as InfoNCA with K=2 and alpha->0, and NCA's self-normalized parameterization keeps absolute likelihoods from decreasing.
Key results
- On UltraFeedback with Mistral-7B, InfoNCA reaches MT-bench 7.63 and AlpacaEval 92.35 versus 7.34 and 90.60 for the DPO-style baseline, and NCA reaches 7.52 MT-bench with 59.4% win rate vs DPO.
- On UltraInteract with Mixtral-8x7B-SFT, NCA raises the average score from 50.1 (DPO) to 57.9, where DPO drops the average below the SFT model's 56.3.
- Ablating the number of responses shows consistent gains from K=2 to K=4 for both InfoNCA and NCA across hyperparameters, indicating suboptimal responses help.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use scalar reward data with InfoNCA instead of preference-based alignment | Mistral-7B · UltraFeedback, MT-bench | score: 7.34 -> 7.63 | Helped |
| use scalar reward data with InfoNCA instead of preference-based alignment | Mistral-7B · UltraFeedback, AlpacaEval | score: 90.60 -> 92.35 | Helped |
| use scalar reward data with NCA instead of preference-based alignment | Mistral-7B · UltraFeedback, MT-bench | score: 7.34 -> 7.52 | Helped |
| increase contrastive responses per prompt from K=2 to K=4 (ablation) | InfoNCA and NCA · UltraFeedback | evaluation performance | Helped |
| use combinatorial DPO×3 rather than simple data pruning | DPO · UltraFeedback | alignment performance | Helped |
| use combinatorial DPO×C_4^2 rather than simple data pruning | DPO · UltraFeedback | alignment performance | Hurt |
| replace likelihood-ratio functional form with log policy likelihood (ablation) | InfoNCA · UltraFeedback, MT-bench | score: 7.63 -> 4.65 | Hurt |
| use log policy likelihood and add SFT regularization (ablation) | InfoNCA · UltraFeedback, MT-bench | score: 7.63 -> 7.15 | Hurt |
| replace likelihood-ratio functional form with log policy likelihood (ablation) | NCA · UltraFeedback, MT-bench | score: 7.52 -> 5.82 | Hurt |
| use log policy likelihood and add SFT regularization (ablation) | NCA · UltraFeedback, MT-bench | score: 7.52 -> 6.88 | Hurt |
| use NCA rather than DPO for pairwise preference alignment | Mistral-7B · UltraInteract | Avg.: 25.5 -> 35.7 | Helped |
| use NCA rather than DPO for pairwise preference alignment | Mixtral-8x7B · UltraInteract | Avg.: 50.1 -> 57.9 | Helped |
Surprising findings
- DPO can hurt reasoning performance on Mixtral-8x7B-SFT (average drops from 56.3 to 50.1), while NCA improves it.
- Naively combining DPO losses over all response pairs (DPO x C_4^2) underperforms simple pruning, performing worse than the baseline.
Limitations
- Theoretical guarantees rely on unlimited model capacity and data, so practical behavior depends on approximations such as estimating Z(x) with the K sampled responses.
- Reward-based experiments use GPT-4-rated UltraFeedback and GPT-4-based evaluation, which may favor methods aligned with GPT-4 judgments.
- Pairwise results on UltraInteract show DPO outperforming NCA on some general instruction tasks, so NCA is not uniformly better.
Experimental setup
Datasets: UltraFeedback UltraInteract
Benchmarks: MT-bench AlpacaEval BBH-Hard HumanEval LeetCode GSM-Plus MATH TheoremQA SVAMP ASDiv
Metrics: MT-bench score AlpacaEval win rate pass rate / accuracy win rate vs DPO KL divergence human preference
Baselines: DPO IPO KTO SLiC-HF Zephyr ORPO
Builds on: DPO InfoNCE Noise Contrastive Estimation Zephyr TRL
Closest papers by meaning
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.93)
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.92)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.92)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.92)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.92)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.