Research map / Reinforcement learning / RL from feedback
Behavior Injection: Preparing Language Models for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BRIDGE, a data augmentation that injects exploration (reflection) and exploitation (subgoal computation, information analysis) behaviors into SFT data so LLMs benefit more from subsequent GRPO RL, evaluated on iGSM and PromptBench.
Problem
LLMs respond very inconsistently to RL finetuning after SFT, with some gaining substantially while others plateau or degrade; the paper seeks to understand and fix this divergence.
Approach
Derives a per-step influence formula for GRPO showing dependence on rollout accuracy (sqrt(alpha(1-alpha))) and data co-influence (NTK-like kernel), then injects exploration (reflection) and exploitation (subgoal computation, information analysis) behaviors into the SFT CoT data to make models "RL-ready".
Key results
- On iGSM with Qwen-1.5B, BRIDGE reaches 91.4% in-distribution and 83.0% OOD after RL, versus 46.2%/36.4% for vanilla RL.
- On PromptBench with Qwen-3B, BRIDGE reaches 85.8% in-distribution and 70.0% OOD after RL, versus 50.0%/34.6% for vanilla.
- BRIDGE gives the largest RL gain across all three base models on both tasks, e.g., +46.6 points in-distribution for Qwen-1.5B on iGSM versus +6.0 for vanilla.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Inject exploration and exploitation behaviors into SFT data before RL | Qwen-1.5B · iGSM In-Dist | RL accuracy gain over SFT: 6.0 -> 46.6 | Helped |
| Inject exploration and exploitation behaviors into SFT data before RL | Qwen-1.5B · iGSM OOD | RL accuracy gain over SFT: 7.4 -> 48.8 | Helped |
| Inject exploration and exploitation behaviors into SFT data before RL | Qwen-1.5B · PromptBench arithmetic In-Dist | RL accuracy gain over SFT: 24.2 -> 53.0 | Helped |
| Inject exploration and exploitation behaviors into SFT data before RL | Qwen-1.5B · PromptBench arithmetic OOD | RL accuracy gain over SFT: 19.6 -> 40.2 | Helped |
| Add subgoal computation behavior (ablation) | Qwen2.5-1.5B BRIDGE · iGSM | RL performance gain | Helped |
| Add information analysis behavior (ablation) | Qwen2.5-1.5B BRIDGE · iGSM | RL performance gain | Helped |
| Add reflection behavior (ablation) | Qwen2.5-1.5B BRIDGE · iGSM | RL performance gain | Helped |
| Use a single injected behavior instead of the combined behaviors (ablation) | Llama3.2-1B BRIDGE · iGSM | RL performance gain | Mixed |
| Inject reflection and analysis behaviors rather than setting their injection probability to p=0 (ablation) | BRIDGE · iGSM | final accuracy | Helped |
| Vary behavior injection probability among non-zero values (ablation) | BRIDGE · iGSM | final accuracy | No clear effect |
Surprising findings
- BRIDGE has lower pre-RL SFT accuracy on PromptBench (e.g., 2.2% vs 13.6% for Qwen-1.5B) yet achieves the best final RL performance.
- Reflection increases policy entropy while subgoal computation and analysis reduce perplexity, indicating separate exploration and exploitation effects.
Limitations
- Evaluation limited to synthetic math and arithmetic/logical reasoning tasks (iGSM, PromptBench); broader agentic domains untested.
- Modified iGSM (removed modulo, filtered answers) departs from the original benchmark.
- Behavior extraction relies on string matching for structured CoT or an oracle LLM for unstructured CoT; potential misuse of behavior injection noted.
Experimental setup
Datasets: iGSM PromptBench
Benchmarks: iGSM PromptBench
Metrics: strict match accuracy RL gain over SFT (delta)
Baselines: Vanilla SFT PP-Aug RC-Aug
Builds on: GRPO DeepSeekMath iGSM PromptBench Dr.GRPO
Closest papers by meaning
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.91)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.91)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.91)
- Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.91)
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.90)
- Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.