Research map / Large language models / Reasoning and chain of thought

DeCoRL: Decoupling Reasoning Chains via Parallel Sub-Step Generation and Cascaded Reinforcement for Interpretable and Scalable RLHF

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeCoRL, which decomposes reasoning into parallel sub-step modules with local and counterfactual contribution rewards and a phased DRPO training scheme, claiming state-of-the-art reward-model results on RM-Bench and RMB.

Problem

Chain-of-thought RL for reward modeling/RLHF gives undifferentiated, monolithic reward signals that hinder error attribution, and sequential decoding is slow for real-time deployment.

Approach

Splits reasoning into k specialized sub-models that run concurrently, scores each sub-step with a local reward plus a counterfactual contribution reward (leave-one-module-out), and trains them with a two-phase "Cascaded DRPO" that combines module-wise and joint GRPO updates.

Key results

• 32B model reaches 80.8 average on RM-Bench vs 70.1 for the best listed baseline (Skywork-Reward-Llama-3.1-8B), per the paper's table.
• On RMB, the 32B model scores 0.757 overall, above GPT-4o (0.738).
• Paper reports 3.8x latency reduction (1,202ms to 316ms) on 10-step problems; the RewardBench results are said to be in an appendix not included in the text.

Surprising findings

• Sequential execution gives nearly the same accuracy as parallel execution (80.5 vs 80.8 on RM-Bench), so the efficiency gain comes without accuracy loss in the authors' ablation.
• Reported speedup is 3.8x in the abstract but the ablation text says sequential execution increased latency 3.7x while the table shows 1,172ms (+271%), an internal inconsistency.

Limitations

• Many headline numbers (interpretability 22.7%, energy, throughput, scalability) lack described measurement protocols and appear inconsistent across sections.
• Method description is vague: the DRPO loss is not well-defined, the contribution reward uses the same reward model whose outputs it scores, and the modules and interfaces are not clearly specified.
• RewardBench results are deferred to an appendix that is not included in the text.

Experimental setup

Datasets: MATH OffsetBias UltraFeedback HelpSteer2-Preference Skywork Reward Preference 80K Code-Preference-Pairs Math-DPO-10K

Benchmarks: RM-Bench RMB RewardBench

Metrics: accuracy average score latency throughput energy consumption interpretability

Baselines: Skywork-Reward-Llama-3.1-8B GPT-4o Eurus-RM-7b Nemotron-340B-Reward DeepSeek-GRM-27B Claude-3-5-sonnet Starling-RM-34B Qwen2-72B-Instruct

Builds on: GRPO DPO Chain-of-Thought Tree-of-Thought Math-Shepherd

Closest papers by meaning

• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.88)
• RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.88)
• DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains (ICLR 2026 · similarity 0.88)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners (ICLR 2026 · similarity 0.88)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.88)