Research map / Large language models / Reasoning and chain of thought

Parallel-R1: Towards Parallel Thinking via Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Parallel-R1, an RL framework that teaches LLMs to think in parallel via a progressive curriculum (SFT cold start on GSM8K-derived parallel traces, then GRPO on DAPO), with causal and structured-attention variants.

Problem

Training LLMs to adopt parallel thinking (exploring multiple reasoning paths concurrently) is hard because SFT on synthetic data encourages imitation rather than exploration, and RL lacks parallel-thinking trajectories to learn from (cold-start problem) on complex math.

Approach

Use a cheap prompt-generated parallel-thinking cold start on easy GSM8K problems, then RL on harder DAPO math with accuracy-based rewards (and an alternating accuracy/parallel reward for the structured variant); also reuse parallel-thinking RL as a mid-training exploration scaffold.

Key results

• Parallel-R1-Seen reaches average 48.9 vs 45.1 for GRPO (DAPO) baseline across AIME25/AIME24/AMC23/MATH.
• Mid-training exploration scaffold variant peaks at 25.6% on AIME25, a 42.9% relative improvement over baseline per the abstract.
• Parallel-SFT-Seen data pipeline: 83.7% valid parallel format on GSM8K vs 0.0% on DAPO with the same prompts.

Surprising findings

• Directly rewarding parallel structure degrades accuracy (e.g., AMC23 drops to 59.4), while accuracy-only reward yields a low parallel ratio of 13.6.
• The structured attention variant worsens when stage-one RL on GSM8K is added (-8.6% average), contrary to the causal variant.

Limitations

• Evaluated only on math benchmarks; RL on toy/synthetic tasks in related work is not extended to other domains.
• Structured attention variant underperforms the causal variant and is sensitive to training recipe.

Experimental setup

Datasets: GSM8K DAPO

Benchmarks: AIME25 AIME24 AMC23 MATH

Metrics: mean@16 accuracy pass@16 mean@1 on MATH parallel ratio

Baselines: GRPO (DAPO) GRPO on GSM8K then DAPO Parallel-SFT-Seen Parallel-SFT-Unseen

Builds on: GRPO DAPO VERL Qwen3

Closest papers by meaning

• PointThinker: Point-Incentivized Parallel Thinking for Multimodal Large Language Model (CVPR 2026 · similarity 0.90)
• ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models (ICML 2026 · similarity 0.90)
• RLP: Reinforcement as a Pretraining Objective (ICLR 2026 · similarity 0.89)
• RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.89)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.89)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.89)