Research map / Large language models / Agents, code and math
Teaching Models to Improve on Tape
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains Llama-2-7B-chat and Llama-3-8B-instruct with PPO over simulated multi-attempt critique sessions (reward = best critique score) so they learn to use corrective feedback, improving controlled generation and transferring to unseen tasks.
Problem
LLMs often fail at controlled generation under constraints that are easy to verify, and do not effectively use corrective feedback from an automated critique across multiple attempts; training data with labeled correct outputs is assumed unavailable.
Approach
CORGI simulates multi-turn interaction sessions with an automated critique that returns textual feedback and a score, and optimizes the policy with PPO using the best score in each session as reward, with feedback tokens masked out of the policy update, so the model learns to exploit feedback and generalizes this skill to unseen tasks.
Key results
- Llama-2 multi-task CORGI reaches 67.5 overall success vs 62.8 few-shot vanilla and 62.5 multi-task RL-NoFB.
- Llama-3 single-task CORGI reaches 88.9 overall success vs 84.3 vanilla and 85.6 single-task RL-NoFB.
- Multi-task CORGI improves target-task success over single-task CORGI by 4 points for Llama-2 (54.5 to 58.5), indicating meta-learning transfer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with iterative critique feedback instead of feedback-free PPO | Llama-2-7B-Chat, single-task training · Source tasks | success rate: 71.4 -> 73.7 | Helped |
| Train with iterative critique feedback instead of feedback-free PPO | Llama-2-7B-Chat, single-task training · Target tasks | success rate: 53.9 -> 54.5 | Helped |
| Train with iterative critique feedback instead of feedback-free PPO | Llama-2-7B-Chat, multi-task training · Overall | success rate: 62.5 -> 67.5 | Helped |
| Train with iterative critique feedback instead of feedback-free PPO | Llama-3-8B-Instruct, single-task training · Source tasks | success rate: 87.6 -> 92.8 | Helped |
| Train with iterative critique feedback instead of feedback-free PPO | Llama-3-8B-Instruct, multi-task training · Target tasks | success rate: 84 -> 85.2 | Helped |
| Use multi-task rather than single-task CORGI training | Llama-2-7B-Chat · Target tasks | success rate: 54.5 -> 58.5 | Helped |
| Use multi-task rather than single-task CORGI training | Llama-3-8B-Instruct · Source tasks | success rate: 92.8 -> 90.8 | Hurt |
| Replace detailed critique with generic binary feedback (ablation) | CORGI, Llama-2-7B-Chat · Numerical Planning | success rate | Hurt |
| Add iterative training with binary critique feedback (ablation) | Vanilla Llama baseline · Numerical Planning | success rate | Helped |
| Move “Output:” outside the user instruction and end the user message before it | CORGI output template · Controlled generation tasks | performance | Helped |
Surprising findings
- RL-NoFB, trained without feedback, gives no gain over vanilla on target tasks for Llama-2 and does not improve with multi-task training.
- Replacing detailed feedback with binary feedback markedly reduces CORGI's gains, though it still beats vanilla Llama.
Limitations
- Limited by context length since feedback history must fit in the window.
- Requires training and updating model weights, impractical for larger models like 70B or commercial models.
- Relies on tasks where good automated critiques exist; suboptimal for creative writing.
Experimental setup
Datasets: Amazon US Customer Reviews ROCStories ECQA Reddit Short Stories QQP-Pos Wiki Neutrality Corpus CommonGen-lite MBPP CommonGen-Hard Program Synthesis numeric subset
Benchmarks: Sentiment Reviews Generation Story Generation Rationale Generation Numerical Planning Controlled Paraphrase Generation Style Transfer Clustering Panagram CommonGen-lite Program Synthesis MBPP CommonGen-Hard
Metrics: success rate
Baselines: Vanilla-Llama RL-NoFB
Builds on: PPO RLHF TRL
Closest papers by meaning
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.89)
- SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.88)
- Reward-Guided Prompt Evolving in Reinforcement Learning for LLMs (ICML 2025 · similarity 0.88)
- RLVF: Learning from Verbal Feedback without Overgeneralization (ICML 2024 · similarity 0.88)
- Language Model Self-improvement by Reinforcement Learning Contemplation (ICLR 2024 · similarity 0.88)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.