Research map / Reinforcement learning / RL from feedback
Reasoning Gym: Reasoning Environments for Reinforcement Learning with Verifiable Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Reasoning Gym, a library of over 100 procedurally generated, algorithmically verifiable reasoning environments for RLVR, and shows difficulty cliffs, intra/cross-domain transfer, and curriculum gains with Qwen2.5-3B-Instruct.
Problem
Reasoning model RLVR training is bottlenecked by fixed, scarce, or memorized datasets, and existing benchmarks lack controllable difficulty and unlimited verifiable instances.
Approach
Procedural generators with parametric difficulty, structural and stylistic parameters, and automatic verifiers across 100+ reasoning tasks, enabling infinite training data, dynamic curricula, and continuous difficulty-based evaluation.
Key results
- Reasoning models o3-mini (63.5%), DeepSeek-R1 (59.5%) and Grok 3 Mini (55.1%) lead zero-shot averages, versus Llama 4 Maverick (41.5%) and Claude 3.5 Sonnet (40.3%) among non-reasoning models.
- RG-Math RLVR on Qwen2.5-3B-Instruct improves MATH from 48.5 to 58.2 and BBH from 8.68 to 16.34, with GSM8K nearly unchanged (76.2 to 76.7).
- Curriculum RLVR outperforms non-curriculum training in all tested environments, e.g. +40.67 points on Spell Backwards at word length 4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| RL-fine-tune on algebra tasks | Qwen2.5-3B-Instruct · Algebra (intra-domain) | Acc@3: 5.0 -> 16.7 (+11.7) | Helped |
| RL-fine-tune on algorithmic tasks | Qwen2.5-3B-Instruct · Algorithmic (intra-domain) | Acc@3: 52.3 -> 59.7 (+7.4) | Helped |
| RL-fine-tune on arithmetic tasks | Qwen2.5-3B-Instruct · Arithmetic (intra-domain) | Acc@3: 89.7 -> 96.0 (+6.3) | Helped |
| RL-fine-tune on cognition tasks | Qwen2.5-3B-Instruct · Cognition (intra-domain) | Acc@3: 40.3 -> 42.3 (+2.0) | Helped |
| RL-fine-tune on games tasks | Qwen2.5-3B-Instruct · Games (intra-domain) | Acc@3: 0.0 -> 3.3 (+3.3) | Helped |
| RL-fine-tune on algorithmic tasks | Qwen2.5-3B-Instruct · Algebra (cross-domain) | Acc@3: 23.83 -> 52.89 (+29.1) | Helped |
| RL-fine-tune on algebra tasks | Qwen2.5-3B-Instruct · ARC | Acc@3: 6.49 -> 4.18 (−2.3) | Hurt |
| RL-fine-tune on math tasks | Qwen2.5-3B-Instruct · MATH (0-shot, CoT) | Score: 48.5 -> 58.2 (+9.7) | Helped |
| RL-fine-tune on math tasks | Qwen2.5-3B-Instruct · GSM8K (8-shot, CoT) | Score: 76.2 -> 76.7 (+0.5) | Helped |
| RL-fine-tune on math tasks | Qwen2.5-3B-Instruct · Big-Bench Hard (3-shot, CoT) | Score: 8.68 -> 16.34 (+7.66) | Helped |
| RL-fine-tune on algorithmic tasks | Qwen2.5-3B-Instruct · MMLU-Pro Math | Score: 54.63 -> 53.89 (−0.74) | Hurt |
| Use progressive curriculum instead of uniform sampling across difficulty levels | Qwen2.5-3B-Instruct trained on Spell Backwards · Spell Backwards, word length 4 | Score: 30.00 -> 70.67 (+40.67) | Helped |
Surprising findings
- Games-trained RLVR yields cross-domain gains on algebra (+21.8) despite the base model scoring zero on games in-domain.
- Algorithmic-task RLVR improves algebra by +29.1 and geometry by +22.3 points on held-out tasks.
Limitations
- Procedural generators struggle with domains requiring domain knowledge or creativity and unstructured answers.
- Single-turn, text-only tasks; no multi-turn or multimodal reasoning yet.
- Uniform i.i.d. task sampling; continual learning and forgetting not studied.
Experimental setup
Datasets: Reasoning Gym tasks (algebra, arithmetic, algorithmic, cognition, games, geometry, logic, graphs) GSM8K MATH Big-Bench Hard MMLU-Pro
Benchmarks: Reasoning Gym tasks (zero-shot easy/hard) GSM8K MATH Big-Bench Hard MMLU-Pro
Metrics: accuracy Acc@3 standard error
Baselines: Qwen2.5-3B-Instruct o3-mini DeepSeek-R1 Llama 4 Maverick Claude 3.5 Sonnet Gemma 3 27B Grok 3 Mini
Builds on: GRPO RLVR Tülu 3
Closest papers by meaning
- PUZZLES: A Benchmark for Neural Algorithmic Reasoning (NeurIPS 2024 · similarity 0.89)
- AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.88)
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.88)
- RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.88)
- The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.