Research map / Large language models / Reasoning and chain of thought

Agentic Multi-Turn Reasoning: A Fairness Approach

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Φ-MPO, a multi-level, focal-weighted trajectory preference optimization method for training the planner of a multi-turn tool-using agent (AgentFlow-style), reporting gains over Flow-GRPO on search, agentic, math and science benchmarks.

Problem

Training multi-turn agentic LLM systems is hard due to long-horizon credit assignment under final-outcome-only supervision and imbalanced data distributions that bias trajectory-level optimization toward dominant patterns.

Approach

Combines trajectory-level DPO with a multi-level preference loss over aligned intermediate states of preferred/dispreferred rollouts (aligned by DP), and adds a focal weight (1-p)^gamma to down-weight easy pairs; the authors argue DPO and GRPO share the same pairwise gradient direction.

Key results

• Phi-MPO with Qwen2.5-7B-Instruct reaches 81.6% on Bamboogle, 81.5% on 2Wiki, 41.7% on GAIA and 50.0% on AIME24, versus 69.6%, 77.2%, 33.1%, 40.0% for Flow-GRPO.
• Ablation: removing the multi-level and focal terms drops Bamboogle from 81.6% to 61.6% (DPO only).
• Phi-MPO with Qwen3.5-9B reports 85.6% on Bamboogle and 49.6% on GAIA.

Surprising findings

• The paper's ablation shows that reducing the multi-level weight to 0.5 degrades performance more than reducing the DPO weight, which the authors take to mean the fair multi-level objective matters most.

Limitations

• Performance depends on hyperparameters (gamma, loss weights, rollout budget) which require tuning.
• Relies on quality of on-the-fly rollouts and LLM-judge reward reliability.
• Evaluation with a GPT-4o judge and a noted tie/exclusion rule for preference construction.

Experimental setup

Datasets: Search-R1 training data DeepMath

Benchmarks: Bamboogle 2Wiki HotpotQA Musique GAIA AIME2024 AMC23 GameOf24 GPQA MedQA

Metrics: accuracy

Baselines: Flow-GRPO AgentFlow Search-R1 GPT-4o GPT-4o-mini SFT Iter-RetGen ZeroSearch ReSearch StepSearch VerlTool AutoGen

Builds on: DPO GRPO Flow-GRPO AgentFlow Search-R1

Closest papers by meaning

• Building Math Agents with Multi-Turn Iterative Preference Learning (ICLR 2025 · similarity 0.89)
• Rectify Evaluation Preference: Improving LLMs’ Critique on Math Reasoning via Perplexity-aware Reinforcement Learning (AAAI 2026 · similarity 0.88)
• Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.88)
• From Debate to Equilibrium: Belief‑Driven Multi‑Agent LLM Reasoning via Bayesian Nash Equilibrium (ICML 2025 · similarity 0.88)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.88)
• Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation (ICLR 2026 · similarity 0.88)