Research map / Reinforcement learning / RL from feedback

DAPO : Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DAPO, an offline step-level RL method that trains a value critic on Monte Carlo completion estimates and then optimizes the LLM policy with a squared-loss advantage-regression objective, evaluated on math and code benchmarks.

Problem

Sparse terminal rewards and instability of actor-critic methods (PPO) when improving multi-step reasoning in LLMs, with high sampling cost for on-policy RL.

Approach

Trains a step-level critic separately (BCE on MC estimates of success probability) and then fits the policy offline to critic-estimated advantages via a squared loss derived from KL-regularized step-level RL, avoiding co-training of actor and critic.

Key results

• Skywork-Math-LLama improves on MATH from 41.90 to 46.88 after one DAPO iteration and 50.54 after iteration 2.
• Qwen2.5-Math-7B-Instruct improves on MATH from 83.42 to 84.86 with DAPO.
• META-LLama-3.1-8B-Instruct gains +3.0 on HumanEval and +9.2 on MBPP+ but OpenO1-LLama-8B-v0.1 drops 2.4 on LiveCodeBench.

Measured effects reported in the paper

Surprising findings

• DAPO decreases performance on some out-of-domain benchmarks, e.g., Minerva MATH for META-LLama-3.1-8B-Instruct (-2.94) and Qwen2-Math-7B-Instruct (-2.56).
• The critic is deliberately treated as a value function rather than a process reward model, arguing that using it as a reward risks reward hacking on step count.

Limitations

• Only a small number of training queries (7500 MATH, ~4000 TACO) used; scaling behavior unexplored.
• Iterative DAPO only tested on two models and two iterations due to compute limits.
• Computational cost of the full pipeline is high.

Experimental setup

Datasets: MATH TACO

Benchmarks: MATH GSM8K Minerva Math OlympiadBench College Math HumanEval HumanEval+ MBPP MBPP+ LiveCodeBench

Metrics: greedy accuracy (pass@1, zero-shot, temperature 0)

Baselines: Qwen2.5-Math-7B-Instruct Qwen2-Math-7B-Instruct Skywork-O1-Open-LLama3.1-8B META-LLama-3.1-8B-Instruct OpenO1-LLama-8B-v0.1

Builds on: DPO DRO PPO GRPO Step-DPO

Closest papers by meaning

• DAPO: An Open-Source LLM Reinforcement Learning System at Scale (NeurIPS 2025 · similarity 0.91)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.91)
• Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.91)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.90)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.89)
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.89)