Research map / Large language models / Alignment and preferences
Language Model Self-improvement by Reinforcement Learning Contemplation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SIRLC, an unsupervised method where an LLM acts as both student (answers unlabeled questions) and teacher (self-evaluates answers), with PPO optimizing the self-evaluation reward; applied to FLAN-T5 on BigBench-hard, IWSLT 2017 and CNN/DailyMail.
Problem
Fine-tuning LLMs usually requires labeled data or human feedback; the paper asks whether an LLM can improve itself using only its own evaluations of generated text, without external labels or reward models.
Approach
Exploit the observation that LLMs are better at evaluating text than generating it: the same frozen initial LLM scores its own answers (correctness prompt for reasoning, 1-10 quality prompt for translation/summarization), and the score is used as an RL reward via PPO to update the policy.
Key results
- SIRLC improves average accuracy over direct generation (DG) by 5.6% across 12 BigBench-hard tasks (training curves and Table with SC, LMSI, RLFT baselines).
- On IWSLT 2017 translation, BERTScore rises from 0.818 to 0.86; on CNN/Daily Mail summarization from 0.886 to 0.899.
- Self-evaluation accuracy exceeds generation accuracy at all FLAN-T5 scales, by about 15% for FLAN-T5-Large/XL on CommonGen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| RL training with self-evaluation reward (SIRLC) versus direct generation | FLAN-T5-Large on BigBench-hard · BigBench-hard 12 tasks | answer accuracy: +5.6% average accuracy over DG | Helped |
| RL training with self-evaluation reward (SIRLC) versus direct generation | FLAN-T5-Large on translation · IWSLT 2017 zh-en | BERTScore: 0.818 to 0.86 | Helped |
| RL training with self-evaluation reward (SIRLC) versus direct generation | FLAN-T5-Large on summarization · CNN/Daily Mail | BERTScore: 0.886 to 0.899 | Helped |
| Training with self-evaluation reward vs. oracle-label RL (RLFT) | FLAN-T5-Large on BigBench-hard · Web of Lies, Logical Deduction (7), Navigate, Penguins | answer accuracy: e.g. Web of Lies 52.9% vs RLFT 72.2% | Hurt |
| Self-evaluation (check answer and regenerate) at inference, no training (ablation) | FLAN-T5-Large on BigBench-hard · 27 BigBench tasks average | answer accuracy: 31.6% to 32.9% | Helped |
| Training on five BigBench tasks then testing on unseen tasks (ablation) | FLAN-T5-Large SIRLC-trained model · unseen BigBench tasks | average answer accuracy: +0.8 average; +1.5, +0.7, +2.2 on three tasks; -0.4 and -0.1 on two | Mixed |
| Self-evaluation reward size scaling (FLAN-T5 Small/Base/Large) (ablation) | FLAN-T5 80M to 780M · three BigBench tasks | answer accuracy: qualitative, largest gain at 80M | Helped |
Surprising findings
- Self-evaluation with the correctness prompt improved answer accuracy over direct generation on 11 of 12 BigBench tasks without any training.
- LMSI performed worse than DG on several BigBench tasks (e.g., 13.1% vs 35.2% on Logical Deduction (7)).
Limitations
- Requires an unlabeled training dataset of instruction-formatted questions.
- Evaluation only on LLMs up to 780M parameters (FLAN-T5-Large), and the evaluator is the initial model, whose judgments may weaken as the policy improves.
- Some attributes are hard to evaluate, limiting applicability to tasks without reliable self-evaluation.
Experimental setup
Datasets: BigBench-hard IWSLT 2017 CNN/Daily Mail BBC CommonGen
Benchmarks: BigBench-hard (12 tasks) IWSLT 2017 (zh-en) CNN/Daily Mail BBC
Metrics: answer accuracy BERTScore BLEU ROUGE
Baselines: DG (direct generation) Self-consistency (SC) LMSI RLFT (oracle-metric RL)
Builds on: PPO self-consistency LMSI chain-of-thought prompting trlx
Closest papers by meaning
- Training Language Models to Self-Correct via Reinforcement Learning (ICLR 2025 · similarity 0.89)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.89)
- SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.89)
- Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR 2024 · similarity 0.88)
- Enabling Lanuguage Models to Implicitly Learn Self-Improvement (ICLR 2024 · similarity 0.88)
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.