Research map / Multi-agent systems and games / LLM agents

Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies "lazy agent" behavior in sequential meta-thinking/reasoning multi-agent RL (ReMA), attributes it partly to a length-normalization bias in multi-turn GRPO, and proposes Dr. MAMR with Shapley-inspired causal influence and a verifiable restart reward.

Problem

Multi-agent LLM reasoning frameworks trained with multi-turn RL suffer from lazy agents, where one agent contributes little and the system collapses toward a single agent, and the reasoning agent can get lost in long multi-turn interactions.

Approach

Removes the 1/T turn-count normalization in multi-turn GRPO (argued to bias toward fewer turns), estimates step-level causal influence by averaging log-probability changes across semantically similar steps grouped over rollouts, and adds a verifiable restart reward that rewards discarding earlier reasoning when it helps the correct answer.

Key results

• On Qwen2.5-7B-Instruct, Dr. MAMR averages 58.43 across seven math benchmarks versus 55.08 for GRPO and 51.97 for ReMA.
• On Qwen2.5-14B-Instruct, Dr. MAMR averages 62.49 versus 58.05 for GRPO and 57.24 for ReMA.
• Under ReMA the reasoning agent's causal influence decays toward zero during training, while Dr. MAMR keeps both agents' influence increasing.

Surprising findings

• ReMA underperforms single-agent GRPO on math benchmarks, indicating lazy agents erase the benefit of multi-agent collaboration.
• Adding a process reward model for per-turn credit causes training collapse after about 30 steps via reward hacking.

Limitations

• Gains are smaller on 3B and 14B models than on 7B, attributed to weaker instruction-following capability.
• Causal influence relies on a single continuation per step and a semantic-similarity threshold (cosine 0.9) with an external embedding model.
• Restart behavior requires an SFT cold start built with GPT-4o-generated noisy data.

Experimental setup

Datasets: DeepScaleR MATH ReMA meta-thinking and reasoning datasets

Benchmarks: MATH500 GSM8K AIME24 AMC23 Gaokao2023en Minerva Math OlympiadBench AIME25

Metrics: pass@1 pass@K average accuracy causal influence mean reward

Baselines: GRPO VRP (CoT) ReMA

Builds on: ReMA GRPO multi-turn GRPO Dr.GRPO DAPO

Closest papers by meaning

• Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.91)
• Benefits and Limitations of Communication in Multi-Agent Reasoning (ICLR 2026 · similarity 0.90)
• Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs (ICML 2026 · similarity 0.90)
• ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Multi-Agent Debate with Memory Masking (ICLR 2026 · similarity 0.89)
• AgentCDM: Enhancing Multi-Agent Collaborative Decision-Making via ACH-Inspired Structured Reasoning (AAAI 2026 · similarity 0.89)