Research map / Large language models / Instruction and fine-tuning
Recursive Introspection: Teaching Language Model Agents How to Self-Improve
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RISE, an iterative fine-tuning method that teaches LLMs to improve their own answers over multiple turns using on-policy rollouts with oracle or self-generated revisions and reward-weighted regression.
Problem
LLMs fail to improve their responses over sequential attempts at test time (self-correction), even when they hold the needed knowledge; imitation of multi-turn data from stronger models does not teach this capability.
Approach
Convert single-turn problems into a multi-turn MDP, collect on-policy rollouts from the current model, relabel each turn with a best-of-N improved response (from a teacher model or the learner itself), and fine-tune the base model with advantage-weighted regression over all iterations, so it learns to introspect and correct its own mistakes.
Key results
- RISE on Llama2-7B raises 5-turn maj@1 on GSM8K from 10.5 (turn 1 maj@1 baseline) to 55.0 after iteration 2 (+17.7 over its turn-1 score, w/o oracle).
- RISE on Mistral-7B with one iteration improves 5-turn GSM8K maj@1 by 23.9 points (35.3 to 59.2), versus 5.3 for Mistral-7B without RISE.
- GPT-3.5 self-refine without an oracle degrades GSM8K 5-turn performance (66.4 to 61.0), while GPT-3.5 under the RISE-style sequential protocol improves 4.6 points (66.4 to 71.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Llama2-7B with RISE for two iterations instead of using the base model | Llama2-7B · GSM8K | m1@t5: 11.1 -> 55.0 | Helped |
| Train Llama2-7B with RISE for two iterations instead of using the base model | Llama2-7B · MATH | m1@t5: 1.4 -> 10.4 | Helped |
| Train Mistral-7B with RISE for one iteration | Mistral-7B · GSM8K | m1@t5: 39.0 -> 59.2 | Helped |
| Train Mistral-7B with RISE for one iteration | Mistral-7B · MATH | m1@t5: 8.4 -> 18.4 | Helped |
| Use self-distillation training with the learner's own sampled candidates (ablation) | Mistral-7B · GSM8K | m1@t5: 39.0 -> 39.5 | Helped |
| Use self-distillation training with the learner's own sampled candidates (ablation) | Llama-3-8B · GSM8K | m1@t5: 52.5 -> 73.8 | Helped |
| Append a different correct response after each successful rollout (ablation) | RISE Llama2-7B · GSM8K | m1@t5: 50.7 -> 51.1 | Helped |
| Repeat the same correct response after each successful rollout (ablation) | RISE Llama2-7B · GSM8K | m1@t5: 50.7 -> 46.2 | Hurt |
| Train Mistral-7B on RISE rollouts generated by Llama2-7B instead of its own rollouts (ablation) | Mistral-7B · GSM8K | m1@t5: 59.2 -> 62.7 | Helped |
| Train Llama2-7B on RISE rollouts generated by Mistral-7B instead of its own rollouts (ablation) | Llama2-7B · GSM8K | m1@t5: 50.7 -> 45.2 | Hurt |
| Train on oracle answers after intermediate failed attempts rather than only presenting the answer directly (ablation) | RISE · GSM8K | performance | Helped |
| Use reward-weighted training on all rollout data rather than filtering to successful data (ablation) | RISE · GSM8K | later-turn performance | Helped |
Surprising findings
- Self-Refine prompting degrades performance across models without an oracle, including GPT-3.5.
- Single-turn SFT on the same oracle data does not instill self-improvement and sometimes degrades sequential performance.
Limitations
- Only one or two training iterations were feasible due to computational constraints.
- Self-distillation variant gave only one iteration and did not outperform best-of-5 single-turn sampling.
- Evaluated only on math reasoning tasks (GSM8K, MATH, SVAMP).
Experimental setup
Datasets: GSM8K MATH SVAMP
Benchmarks: GSM8K MATH SVAMP
Metrics: maj@1 at turn 5 (m1@t5) pass@1 with oracle at turn 5 (p1@t5) maj@5 at turn 1 (m5@t1) accuracy
Baselines: Self-Refine GLoRE V-STaR Eurus-7B-SFT GPT-3.5 SFT on oracle data self-consistency
Builds on: DAgger reward-weighted regression advantage-weighted regression self-consistency Self-Refine
Closest papers by meaning
- Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.89)
- Self-Challenging Language Model Agents (NeurIPS 2025 · similarity 0.88)
- $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.88)
- LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.88)
- Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains (ICLR 2025 · similarity 0.88)
- $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.