AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO

Yiming Ren, Yiran Xu, Zicheng Lin, Chufan Shi, Yukang Chen, Dingdong WANG, Tianhe Wu, Junjie Wang, Yujiu Yang, Yu Qiao, Ruihang Chu

ICML 2026 · Reinforcement learning › RL from feedback · approach: GRPO stability and variants · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
sample part of each GRPO group from a frozen 1.7B model, annealed to on-policyQwen3-8B-Base GRPO · AIME24Pass@1 (%): 15.0 -> 23.8Improved
sample part of each GRPO group from a frozen 1.7B model, annealed to on-policyQwen3-8B-Base GRPO · AIME25Pass@1 (%): 12.1 -> 22.5Improved
sample part of each GRPO group from a frozen 4B model, annealed to on-policyQwen3-14B-Base GRPO · AIME24, AIME25, MATH-500, OlympiadBenchPass@1 (%): AIME24 18.0 -> 24.4; AIME25 12.9 -> 14.6Improved
sample part of each GRPO group from a frozen 1.8B model, annealed to on-policyInternLM2.5-7B GRPO · AIME24, AIME25, MATH-500, OlympiadBenchPass@1 (%): MATH-500 18.6 -> 22.6Improved
S2L-PO training on math, evaluated out of domainQwen3-8B-Base · CommonsenseQAaccuracy (%): 63.9 -> 67.8Improved
S2L-PO with 1.7B explorer on out-of-domainQwen3-8B-Base · CommonsenseQAaccuracy (%): 63.9 -> 64.2No clear effect
S2L-PO-4B on 14B out of domainQwen3-14B-Base · CommonsenseQAaccuracy (%): 67.2 -> 70.7Improved
filter small-model rollouts so diversity matches the large model (ablation)S2L-PO 1.7B->8B · AIME24Pass@1 (%): 23.8 -> 14.7Hurt
raise sampling temperature to 1.5 (token-level noise) instead of small-model rollouts (ablation)GRPO · AIME24&25Pass@1Hurt
use small-only rollouts without transitioning to on-policy GRPO (ablation)S2L-PO · AIME24&25Pass@1Hurt
use progressive linear annealing instead of abrupt two-phase switch (ablation)S2L-PO · Qwen3-8B-Base AIMEPass@1Improved
lengthen annealing from 5 to 8 steps (ablation)S2L-PO · Qwen3-8B-Base AIMEPass@1Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy