Research map / Multi-agent systems and games / LLM agents
Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows on 4,181 Omni-MATH problems that a hierarchical planner-executor-reviewer (PER) pipeline has higher reviewer precision than broadcast peer deliberation but much lower uptake of useful critique, explaining its lower accuracy.
Problem
Whether specialized reviewer roles improve repair in multi-agent math reasoning, and why a reviewer's correct critiques may fail to turn into better solver candidates.
Approach
Separates reviewer detection (precision), solver uptake (CouplingRate: whether useful critique changes the next carried-forward candidate), and repair (ReviewerGuidedRepairRate), showing that reviewer precision and critique uptake are empirically separable across PER and broadcast protocols.
Key results
- Broadcast reaches 89.2% FinalPassRate vs 85.2% for PER and 78.8% for Single-Agent Iterative on 4,181 problems.
- PER reviewer precision is higher (0.861 vs 0.644), but CouplingRate is 0.336 for PER vs 0.935 for broadcast, and ReviewerGuidedRepairRate is 0.051 vs 0.286.
- Within PER, ACK-required lowers FinalPassRate to 82.5% while EMB recovers it to 86.3%; in a Gemma 3 27B replication PER precision is 0.881 vs broadcast 0.722 while uptake is 0.092 vs 0.742.
Surprising findings
- PER's reviewer has higher precision than broadcast's yet produces far lower uptake of useful critique and lower repair.
- Forcing explicit acknowledgment of critique (ACK-required) lowers final accuracy rather than improving follow-through.
Limitations
- Results are from one primary actor family (gpt-oss-120b) with a reduced Gemma 3 replication on a subset.
- Within-PER interventions are directional rather than causal and do not isolate causal factors; the PER-broadcast comparison confounds multiple protocol differences.
- CouplingRate is an operational answer-transition statistic and not a semantic measure of whether the solver understood the critique.
Experimental setup
Datasets: Omni-MATH Omni-MATH 2
Benchmarks: Omni-MATH
Metrics: FinalPassRate Pass@1 reviewer precision CouplingRate NeglectRate ReviewerGuidedRepairRate AvgEvalCalls/problem Cohen's kappa
Baselines: Baseline LLM Single-Agent Iterative PER (role-based MAS) Broadcast deliberation PER-inner6 ACK-required PER EMB PER
Builds on: Self-Refine Multi-Agent Debate Omni-MATH
Closest papers by meaning
- When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning (NeurIPS 2026 · similarity 0.88)
- Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (ICML 2024 · similarity 0.85)
- Selective Critique for Cost-Aware LLM Agents in Long-Horizon Decision Making (NeurIPS 2026 · similarity 0.84)
- V1: Unifying Generation and Self-Verification for Parallel Reasoners (ICML 2026 · similarity 0.84)
- Agent-as-a-Judge: Evaluate Agents with Agents (ICML 2025 · similarity 0.84)
- NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.