Research map / Reinforcement learning / RL from feedback
Learning to Reason as Action Abstractions with Scalable Mid-Training RL
Shenao Zhang, Donghan Yu, Yihao Feng, Bowen Jin, Zhaoran Wang, John Peebles, Zirui Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| mid-train with RA3 instead of next-token prediction | Llama-3.2-1B · code benchmarks avg | pass@1: 25.3 -> 29.8 | Improved |
| mid-train with RA3 instead of next-token prediction | Qwen2.5-1.5B · code benchmarks avg | pass@1: 41.7 -> 46.6 | Improved |
| mid-train with RA3 instead of next-token prediction | Llama-3.1-8B · code benchmarks avg | pass@1: 47.7 -> 48.9 | Improved |
| mid-train with RA3 instead of next-token prediction | Llama-3.2-1B · code benchmarks avg | pass@5: 40.3 -> 43.1 | Improved |
| mid-train with RA3 instead of next-token prediction | Llama-3.1-8B · MBPP | pass@1: 48.6 -> 48.0 | Hurt |
| mid-train with RA3 instead of next-token prediction | Llama-3.1-8B · MBPP+ | pass@1: 51.1 -> 53.2 | Improved |
| add mid-training (NTP) before RLVR | base models · RLVR post-training | performance | Improved |
| mid-train with RA3 instead of NTP before RLVR | RLVR with GRPO · RLVR post-training | convergence speed and asymptotic performance | Improved |
| use small KL penalty c in latent policy (ablation) | RA3 on Qwen · Qwen mid-training | redundant reasoning at every step | Hurt |
| use large KL penalty c=0.2 (ablation) | RA3 on Qwen · Qwen mid-training | latent behavior: policy outputs only act, RA3 degenerates to NTP | Hurt |
| use default penalty c=0.05 (ablation) | RA3 on Qwen · Qwen mid-training | reasoning frequency: reasoning at fewer than 40% of lines | Improved |
| fine-tune on reasoning-bootstrapped data instead of raw data (ablation) | RA3 M step · mid-training | cross-entropy loss | Improved |
Closest papers by meaning
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.92)
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.90)
- Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.90)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
- Reinforcement Mid-Training (ICLR 2026 · similarity 0.90)
- PRISM: Demystifying Retention and Interaction in Mid-Training (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
