Research map / Large language models / Reasoning and chain of thought

Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GPS, a small generative prompt predictive model conditioned on shared optimization history that predicts prompt difficulty and drives diverse, intermediate-difficulty batch selection for cheaper GRPO post-training of reasoning LLMs.

Problem

RLVR post-training of LLMs is rollout-intensive; online prompt selection helps but evaluation-based methods cost extra rollouts, while prior prompt-specific predictive models fail to generalize across prompts and lag behind policy evolution.

Approach

A lightweight generative prompt predictive model with a global latent "difficulty context" inferred from the full shared optimization history (variational ELBO, history-conditioned prior) predicts difficulty for any prompt, and a greedy max-sum batch acquisition combines intermediate-difficulty utility with history-anchored diversity; the same model is reused for test-time compute allocation.

Key results

• On DSR-1.5B, GPS reaches 56.5 average on math benchmarks, tying DS (Oracle) and above Uniform's 54.9, with 16h runtime versus 30h for DS.
• GPS achieves 1.4x-2.0x training-step speedup over Uniform and up to 69% fewer rollouts than DS, per the paper's claims.
• Test-time difficulty-guided allocation yields up to 3.2% relative gain at fixed budget or 36.4% compute reduction without loss, per the paper.

Surprising findings

• A simple embedding-similarity baseline correlates only about 0.2 with success rates, so semantic embeddings alone are weak difficulty predictors and history context is needed.

Limitations

• Predictive correlation diminishes for semantically distant out-of-domain tasks, suggesting multi-domain co-training may be needed.
• Design choices of the predictor (architecture, variational formulation) are not optimized; the generative architecture is described as a simple instantiation.
• Experiments are limited to math and Countdown tasks with a few model backbones (1.5B-8B).

Experimental setup

Datasets: DeepScaler Countdown-34 Countdown-4 MATH500 OlympiadBench Minerva Math AMC23 AIME24 MMLU-Pro ARC-c GPQA-diamond

Benchmarks: MATH500 OlympiadBench Minerva Math AMC23 AIME24 MMLU-Pro ARC-c GPQA-diamond Countdown-34 Countdown-4

Metrics: pass@1 (avg@k) Spearman rank correlation training runtime rollout cost pass@k

Baselines: Uniform MoPPS PCL GRESO Dynamic Sampling (DS)

Builds on: GRPO MoPPS Dynamic Sampling GRESO verl

Closest papers by meaning

• Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.90)
• GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.88)
• Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.87)
• Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL (ICLR 2024 · similarity 0.87)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.87)
• StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving (NeurIPS 2024 · similarity 0.87)