Research map / Multi-agent systems and games / LLM agents
Reinforce LLM Reasoning through Multi-Agent Reflection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPSDP, an RL-inspired direct preference method that trains an actor-critic LLM pair to iteratively refine math answers, with a theoretical guarantee under coverage assumptions.
Problem
Existing verify-and-improve methods for LLM reasoning use restricted feedback spaces and do not jointly train the actor and critic, leading to suboptimal multi-turn refinement.
Approach
Model multi-turn actor-critic refinement as an episodic MDP with a last-answer-only state and train both agents via a DPO-style loss derived from PSDP-style KL-regularized policy search, using Q-values estimated by rollouts of the reference policy.
Key results
- On MATH 500, DPSDP raises Ministral-8B maj1@turn5 from 57.2% (post-SFT) to 63.2%, with first-turn accuracy 58.2%.
- Llama-3.1-8B-based DPSDP improves MATH 500 maj1@turn5 to 58.4% from 55.8% first-turn accuracy (as reported in intro).
- Ministral-based DPSDP reaches 89.1% maj1@turn5 on GSM8K versus 82.3% after SFT, and 1.2% gain on Olympiad Bench over five attempts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply DPSDP after SFT | Ministral-8B-It · MATH 500 | m1@t5: 57.2 -> 63.2 | Helped |
| apply DPSDP after SFT | Ministral-8B-It · GSM8K | m1@t5: 82.3 -> 89.1 | Helped |
| use one agent for both answering and reflection instead of specialized agents (ablation) | DPSDP · MATH 500 | m1@t5: 63.2 -> 59.6 | Hurt |
| use one agent for both answering and reflection instead of specialized agents (ablation) | DPSDP · Olympiad Bench | m1@t5: 27.0 -> 26.3 | Hurt |
| use one agent for both answering and reflection instead of specialized agents (ablation) | DPSDP · GSM8K | m1@t5: 89.1 -> 90.4 | Helped |
| use one agent for both answering and reflection instead of specialized agents (ablation) | DPSDP · MMLU-Pro Math | m1@t5: 54.2 -> 54.9 | Helped |
| provide full conversation history instead of only the latest answer (ablation) | DPSDP · MATH 500 | m1@t5: 63.2 -> 62.6 | Hurt |
| provide full conversation history instead of only the latest answer (ablation) | DPSDP · GSM8K | m1@t5: 89.1 -> 88.9 | No clear effect |
| remove restart-style data collection (ablation) | DPSDP · MATH 500 | m1@t5: 61.6 -> 63.2 | Hurt |
| remove restart-style data collection (ablation) | DPSDP · GSM8K | m1@t5: 89.5 -> 89.1 | Helped |
| use step-by-step Q-value estimation with Q^{pi_2}_1 in the DPO-style loss (ablation) | DPSDP · mathematical benchmarks | accuracy | No clear effect |
Surprising findings
- The single-agent variant slightly outperforms the multi-agent version on GSM8K and MMLU-Pro Math, suggesting positive transfer between answer generation and feedback.
- Non-Markovian evaluation (full conversation history) degrades performance due to test-time distribution shift.
Limitations
- Relies on ground-truth correctness for rewards and Q-value estimates, limiting applicability to tasks without verifiable answers.
- Practical algorithm deviates from theory (beta set to 0.1, Q-value approximations, one refinement step in training), and the theoretical bound is only loosely tied to practice.
- Evaluated only on math benchmarks with 3B-8B models.
Experimental setup
Datasets: OpenMathInstruct-2 MATH GSM8K
Benchmarks: MATH 500 GSM8K MMLU-Pro Math Olympiad Bench
Metrics: pass1@turn1 maj1@turn5 pass1@turn5 maj5@turn1
Baselines: STaR STaR-DPO Oracle-RISE SFT self-consistency
Builds on: PSDP DPO STaR RISE TRL
Closest papers by meaning
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.92)
- DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs (AAAI 2026 · similarity 0.91)
- ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation (ICLR 2026 · similarity 0.91)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.