Research map / Large language models / Alignment and preferences
Enabling Lanuguage Models to Implicitly Learn Self-Improvement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PIT, which implicitly learns a self-improvement goal from preference data by training a reward model and policy on the gap between a response and a reference response, avoiding hand-written rubrics used by prompting methods like Self-Refine.
Problem
Prompting-based LLM self-improvement requires explicit, carefully written rubrics that are costly and hard to derive for complex goals such as helpfulness and harmlessness.
Approach
Reformulate RLHF so the model maximizes the quality gap of its response conditioned on a reference response, learning the improvement goal implicitly from the same preference data used for reward models, with curriculum RL that first trains on human responses and then on policy-sampled responses.
Key results
- PIT vs. Original win-lose difference of 28.13% (GPT-4) and 14.00% (DeBERTa) on Anthropic/HH-RLHF, and 20.31% (GPT-4) and 7.2% (DeBERTa) on OpenAI/Summary.
- Human evaluation on Anthropic/HH-RLHF prefers PIT over Self-Refine by 23.53%, while GPT-4 prefers Self-Refine by 3.91%.
- Removing the first curriculum RL stage causes PIT to fail to improve original responses (ablation shows negative Delta vs. Self-Refine and PIT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PIT to improve the original response | PaLM 2 (Bison) · Anthropic/HH-RLHF | win rate minus lose rate: 28.13 | Helped |
| Use PIT to improve the original response | PaLM 2 (Bison) · OpenAI/Summary | win rate minus lose rate: 20.31 | Helped |
| Use PIT to improve the original response | PaLM 2 (Bison) · Synthetic Data | win rate minus lose rate: 33.59 | Helped |
| Use PIT instead of Self-Refine | PaLM 2 (Bison) · Anthropic/HH-RLHF | win rate minus lose rate (human evaluation): 23.53 | Helped |
| Use PIT instead of Self-Refine | PaLM 2 (Bison) · Anthropic/HH-RLHF | win rate minus lose rate (GPT-4): -3.91 | Hurt |
| Use PIT instead of Self-Refine | PaLM 2 (Bison) · Anthropic/HH-RLHF | win rate minus lose rate (DeBERTa): 3.70 | Helped |
| Use PIT instead of Self-Refine | PaLM 2 (Bison) · Synthetic Data | win rate minus lose rate (GPT-4): 10.16 | Helped |
| Add the first RL stage before training on policy-model responses (ablation) | PIT · Anthropic/HH-RLHF | win rate minus lose rate vs. PIT (DeBERTa): -30.10 | Helped |
| Add the first RL stage before training on policy-model responses (ablation) | PIT · Anthropic/HH-RLHF | win rate minus lose rate vs. PIT (DeBERTa): -36.70 | Helped |
| Use low improvement temperature (0.4 ~ 0.6) (ablation) | PIT · Anthropic/HH-RLHF | win rate minus lose rate: 0.4 ~ 0.6 | Helped |
| Add unlikelihood loss during PIT supervised fine-tuning (ablation) | PIT · PIT supervised fine-tuning | performance | Hurt |
| Replace pairwise reward-gap training with pointwise MSE reward targets (ablation) | PIT reward model · Held-out reward model test set | accuracy: ~5% accuracy drop | Hurt |
Surprising findings
- GPT-4 judges disagree with human evaluators: GPT-4 prefers Self-Refine over PIT, while humans prefer PIT, attributed to GPT-4's preference for long and detailed responses.
- Self-improvement quality does not increase monotonically with the number of iterations; PIT (Iter 1) is not the best iteration.
Limitations
- Experiments rely on proprietary PaLM 2 models and TPU training, limiting reproducibility; smaller and larger PaLM 2 variants were not used.
- LLM and reward-model evaluators are biased toward their own methods (GPT-4 vs Self-Refine, DeBERTa vs PIT), requiring human evaluation for resolution.
- Improvement iterations do not show a consistent positive trend, so stopping conditions remain unresolved.
Experimental setup
Datasets: Anthropic/HH-RLHF OpenAI/Summary Synthetic instruction-following data
Benchmarks: Anthropic/HH-RLHF OpenAI/Summary Synthetic instruction-following data
Metrics: win rate lose rate win-lose difference (Delta) ELO score agreement with ground-truth labels
Baselines: Self-Refine Original responses y_w in data
Builds on: RLHF Self-Refine PPO context distillation
Closest papers by meaning
- Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.91)
- Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.90)
- Training Language Models to Self-Correct via Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Progress or Regress? Self-Improvement Reversal in Post-training (ICLR 2025 · similarity 0.88)
- Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.88)
- SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.