Research map / Large language models / Alignment and preferences
CREAM: Consistency Regularized Self-Rewarding Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CREAM, a self-rewarding LLM training method that regularizes DPO preference labels by the Kendall-tau consistency of implicit rewards between the current and previous iteration's models.
Problem
Self-rewarding language models generate preference labels with the same LLM acting as policy and judge, so overconfident and noisy labels accumulate across iterations and gains diminish for 7B models.
Approach
Derive a generalized iterative preference fine-tuning framework and show that a regularized DPO equals soft-label DPO, then set the soft label from the Kendall-tau consistency between rankings of the current model and the previous iteration's model, mixing DPO and reversed DPO losses accordingly.
Key results
- On Llama-3 after 3 iterations (M3), CREAM reaches 83.37 average accuracy versus 80.71 for SRLM and 82.12 for the external-reward Oracle.
- On Llama-2 after M3, CREAM reaches 50.51 average accuracy versus 40.08 for SRLM, which degrades across iterations.
- Self-rewarding SRLM on Llama-3 peaks at M2 and degrades by M5-M6 (68.59 average at M6), while CREAM stays near 82.7-83.6 through M6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add consistency-regularized training instead of standard self-rewarding | Llama-3 · ARC-Easy, ARC-Challenge, OpenBookQA, SIQA, GSM8K | exact match accuracy: M3: 87.17 -> 89.52; 81.23 -> 83.36; 87.30 -> 90.20; 70.37 -> 72.06; 77.48 -> 81.73 | Helped |
| add consistency-regularized training instead of standard self-rewarding | Llama-2 · ARC-Easy, ARC-Challenge, OpenBookQA, SIQA, GSM8K | exact match accuracy: M3: 46.55 -> 62.08; 34.47 -> 48.81; 49.20 -> 64.60; 48.06 -> 51.22; 22.14 -> 25.85 | Mixed |
| add consistency-regularized self-rewarding rather than external Oracle reward | Llama-3 · ARC-Easy, ARC-Challenge, OpenBookQA, SIQA, GSM8K | exact match accuracy: 89.31 vs 89.52; 81.31 vs 83.36; 90.20 vs 90.20; 73.75 vs 72.06; 76.04 vs 81.73 | Mixed |
| replace adaptive ranking correlation with a manually set consistency rate (ablation) | CREAM · Llama-3 benchmarks | exact match accuracy: M3 CREAM vs without RC: 89.52 vs 88.09; 83.36 vs 80.55; 90.20 vs 87.20; 72.06 vs 71.39; 81.73 vs 79.23 | Hurt |
| replace adaptive ranking correlation with a manually set consistency rate (ablation) | CREAM · Llama-2 benchmarks | exact match accuracy: M3 CREAM vs without RC: 62.08 vs 60.06; 48.81 vs 49.23; 64.60 vs 65.40; 51.22 vs 49.44; 25.85 vs 25.62 | Mixed |
| use Kendall correlation rather than Spearman consistency (ablation) | CREAM M2 · Llama-3 benchmarks | exact match accuracy: ARC-Easy 86.95 vs 88.89; ARC-Challenge 82.00 vs 80.89; OpenBookQA 85.40 vs 88.00; SIQA 70.05 vs 69.79; GSM8K 78.77 vs 81 | Mixed |
| use Kendall correlation rather than TopOrder consistency (ablation) | CREAM M2 · Llama-3 benchmarks | exact match accuracy: ARC-Easy 87.25 vs 88.89; ARC-Challenge 80.12 vs 80.89; OpenBookQA 86.88 vs 88.00; SIQA 70.83 vs 69.79; GSM8K 79.75 vs 81 | Mixed |
| use Oracle rather than M0 as the baseline reward model for consistency (ablation) | CREAM M2 · Llama-3 ARC-E, ARC-C, OBQA, SIQA, GSM8K | exact match accuracy: 88.89 -> 88.51; 80.89 -> 81.06; 88.00 -> 86.20; 69.79 -> 72.21; 81.04 -> 79.91 | Mixed |
| use Oracle rather than M0 as the baseline reward model for consistency (ablation) | CREAM M2 · Llama-2 ARC-E, ARC-C, OBQA, SIQA, GSM8K | exact match accuracy: 58.97 -> 62.42; 47.53 -> 48.72; 62.80 -> 66.00; 50.43 -> 51.13; 24.41 -> 22.52 | Mixed |
| add consistency-based regularization rather than standard SRLM training | Llama-3 · reward ranking across iterations | Kendall tau: M2 vs M1: -0.22 ± 0.41 -> 0.46 ± 0.35; M3 vs M2: -0.08 ± 0.38 -> 0.84 ± 0.19 | Helped |
| continue iterative self-rewarding with consistency regularization rather than standard SRLM | Llama-3 · GSM8K, M2 to M6 | exact match accuracy: CREAM: 81.04, 81.73, 82.26, 81.73, 82.49; SRLM: 78.01, 77.48, 75.83, 64.22, 59.29 | Mixed |
| use prompt-based LLM-as-a-Judge rewarding instead of intrinsic DPO rewarding (ablation) | 7B self-rewarding models · downstream tasks | exact match accuracy | Hurt |
Surprising findings
- Prompt-based LLM-as-a-Judge rewarding degrades performance already at the first iteration for 7B models, so the implicit DPO reward is used instead.
- Simply weighting the DPO loss (SRLM + Weight) does not match CREAM; negative weights cause catastrophic forgetting.
Limitations
- Requires full fine-tuning across multiple iterations, which is time-consuming and compute-intensive.
- Evaluated only on 7B-scale models (Llama-2, Llama-3) and a few downstream tasks.
- Overall ranking accuracy of self-generated preferences remains low (at most about 70%).
Experimental setup
Datasets: Open Assistant ARC-Easy ARC-Challenge OpenBookQA SIQA GSM8K RewardBench
Benchmarks: ARC-Easy ARC-Challenge OpenBookQA SIQA GSM8K RewardBench
Metrics: exact match accuracy average accuracy win/tie rate pairwise ranking accuracy Kendall tau Spearman TopOrder consistency rate
Baselines: SRLM SRLM + KL Oracle (InternLM2 reward model) P-SRLM (LLM-as-a-Judge) CREAM w/o RC
Builds on: self-rewarding language models direct preference optimization RLHF RLAIF
Closest papers by meaning
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.91)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.89)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.89)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)
- Bootstrapping Language Models with DPO Implicit Rewards (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.