Research map / Large language models / Agents, code and math
Building Math Agents with Multi-Turn Iterative Preference Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-turn direct preference learning (M-DPO and M-KTO) with online iterative training for tool-integrated (Python interpreter) math reasoning, masking external-observation tokens from the loss.
Problem
Improving LLM math problem solving with external tools requires multi-turn decision making; existing work relies on SFT over synthetic data, and single-turn preference methods treat tool outputs as model-generated text, which hurts reasoning.
Approach
Formulate tool-integrated reasoning as a KL-regularized MDP, derive a multi-turn DPO loss (M-DPO) and multi-turn KTO (M-KTO) that sum log-ratios over model-generated turns only (masking external messages, valid under deterministic tool outputs), and run them in an online iterative framework with mixture sampling and reference-model updates.
Key results
- Iterative M-DPO raises Gemma-1.1-it-7B from 77.5% to 83.9% on GSM8K and from 46.1% to 51.2% on MATH after three iterations.
- Iterative M-DPO on Gemma-2-it-9B reaches 86.3% on GSM8K and 54.5% on MATH, from 84.1% and 51.0% for SFT.
- Multi-turn M-DPO/M-KTO consistently outperform RAFT and single-turn iterative DPO/KTO across four base models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply iterative M-DPO instead of SFT | Gemma-1.1-it-7B · GSM8K test | accuracy: 77.5 -> 83.9 | Helped |
| Apply iterative M-DPO instead of SFT | Gemma-1.1-it-7B · MATH test | accuracy: 46.1 -> 51.2 | Helped |
| Use multi-turn rather than single-turn iterative DPO | Gemma-1.1-it-7B · GSM8K test | accuracy: 81.7 -> 83.9 | Helped |
| Use multi-turn rather than single-turn iterative DPO | Gemma-1.1-it-7B · MATH test | accuracy: 48.9 -> 51.2 | Helped |
| Set KL coefficient eta to 0.01 rather than 0.1 (ablation) | Gemma-1.1-it-7B iterative M-DPO · GSM8K test | accuracy: 83.9 -> 81.7 | Hurt |
| Set KL coefficient eta to 0.01 rather than 0.1 (ablation) | Gemma-1.1-it-7B iterative M-DPO · MATH test | accuracy: 51.2 -> 50.1 | Hurt |
| Set KL coefficient eta to 0.5 rather than 0.1 (ablation) | Gemma-1.1-it-7B iterative M-DPO · GSM8K test | accuracy: 83.9 -> 82.8 | Hurt |
| Set KL coefficient eta to 0.5 rather than 0.1 (ablation) | Gemma-1.1-it-7B iterative M-DPO · MATH test | accuracy: 51.2 -> 49.7 | Hurt |
| Keep the reference fixed to the SFT policy instead of updating it each iteration (ablation) | Gemma-1.1-it-7B iterative M-DPO · GSM8K test | accuracy: 83.9 -> 79.9 | Hurt |
| Keep the reference fixed to the SFT policy instead of updating it each iteration (ablation) | Gemma-1.1-it-7B iterative M-DPO · MATH test | accuracy: 51.2 -> 48.0 | Hurt |
| Use mixture sampling rather than only on-policy sampling (ablation) | Gemma-1.1-it-7B iterative preference learning · MATH test | accuracy | Helped |
| Increase trajectories per prompt from N=12 to N=30 (ablation) | Gemma-1.1-it-7B iterative preference learning · test accuracy | accuracy | No clear effect |
Surprising findings
- Preference learning improves pass@n only for small n; for n>16 all models perform similarly.
- Fixing the reference model to the SFT policy performs much worse than updating the reference each iteration (79.9% vs 83.9% GSM8K for Gemma-1.1-7B).
Limitations
- M-DPO loss is only theoretically valid under deterministic tool observations; stochastic environments need a value network and Monte Carlo correction left for future work.
- Gains are concentrated on top of a well-trained SFT model and improve pass@n only for small n.
- Evaluated only on GSM8K and MATH with 7-9B models; no process-level reward or AI-feedback verification used.
Experimental setup
Datasets: Open-MathInstruct MATH train GSM8K train MetaMathQA MMIQC
Benchmarks: GSM8K MATH
Metrics: accuracy pass@n
Baselines: SFT (tool-integrated) RAFT Iterative single-turn DPO Iterative single-turn KTO WizardMath CodeLLaMA-2
Builds on: DPO KTO Iterative DPO (online RLHF) RAFT Open-MathInstruct
Closest papers by meaning
- Rectify Evaluation Preference: Improving LLMs’ Critique on Math Reasoning via Perplexity-aware Reinforcement Learning (AAAI 2026 · similarity 0.90)
- Preference Optimization for Reasoning with Pseudo Feedback (ICLR 2025 · similarity 0.89)
- Agentic Multi-Turn Reasoning: A Fairness Approach (NeurIPS 2026 · similarity 0.89)
- Advancing LLM Reasoning Generalists with Preference Trees (ICLR 2025 · similarity 0.88)
- Iterative Reasoning Preference Optimization (NeurIPS 2024 · similarity 0.88)
- Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.