Research map / Large language models / Reasoning and chain of thought

h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Composes short-horizon GSM8K problems into dependent multi-step chains and trains Qwen-2.5-3B Instruct with outcome-only Dr. GRPO under a horizon curriculum, improving long-horizon, harder math, OOD and long-context performance.

Problem

LLMs' reasoning accuracy drops as the reasoning horizon (number of dependent steps) grows, and existing remedies rely on inference-time scaffolding or costly step-level supervision; RL on standard short data saturates quickly.

Approach

Synthetically chain existing short-horizon problems into dependency chains of arbitrary length via adapters, then run stagewise curriculum RL over horizons with outcome-only rewards, with a theoretical SNR argument that curriculum avoids exponential sample complexity.

Key results

• Curriculum RL (Len-5) raises in-domain composed GSM8K accuracy at horizon 4 from 6.70% to 18.77% and horizon 5 from 3.57% to 9.82% on Qwen-2.5-3B Instruct, versus 2.95% and 2.08% for standard RLVR at those horizons.
• AIME 2024 accuracy improves from 5.10% to 10.52% (2.06x) after curriculum training on composed GSM8K.
• On ReasoningGym propositional logic, accuracy rises from 22.90% to 47.10%; LongBench-v2 improves from 35.00% to 37.90%.

Surprising findings

• Only-L1 RL improves horizon-1 accuracy but gives no gains at longer horizons, and uniform mixing of horizons performs worse than the instruct model at equal compute.
• Standard RLVR on GSM8K is bounded by the base model at pass@128, whereas the curriculum-trained model exceeds it, suggesting new reasoning paths are learned.

Limitations

• Theory uses a simplified independent-step correctness model with assumed parameter separation and forgetting-free curriculum.
• Chaining is restricted to serial dependencies on GSM8K atomic problems; the definition of a 'step' is vague.
• Main experiments use only a 3B Instruct model; several gains on harder benchmarks are modest (e.g., MATH-500 +3 points, AIME 2025 small absolute numbers).

Experimental setup

Datasets: GSM8K MATH ReasoningGym LongBench-v2 Hash-hop

Benchmarks: GSM8K composed (in-domain) MATH-500 GSM-Symbolic MMLU-Pro Math AIME 2024 AIME 2025 ReasoningGym LongBench-v2 Hash-hop

Metrics: accuracy pass@k avg@32

Baselines: Instruct model Only-L1 RL Uniform-Mix Only-Long Standard RLVR on GSM8K

Builds on: GRPO Dr. GRPO RLVR GSM8K

Closest papers by meaning

• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs (ICLR 2026 · similarity 0.91)
• Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning (ICLR 2026 · similarity 0.90)
• LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts (ICLR 2026 · similarity 0.90)
• General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.90)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
• SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning (NeurIPS 2025 · similarity 0.90)