Research map / Multi-agent systems and games / LLM agents
ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReMA, a multi-agent RL framework that splits LLM reasoning into a high-level meta-thinking agent and a low-level reasoning agent, trained jointly with RL for math and LLM-as-a-Judge tasks.
Problem
Single-agent RL for LLM reasoning must learn meta-thinking and detailed reasoning within one autoregressive pass, which leads to inefficient exploration, poor readability, poor OOD generalization, and early convergence to local optima.
Approach
Decouple reasoning into a high-level meta-thinking agent that produces strategic plans and a low-level reasoning agent that executes them, and train both with RL under aligned rewards in a multi-agent setup; extend to multi-turn with shared parameters and a turn-level ratio in GRPO.
Key results
- Single-turn ReMA averages 6.68 points over CoT for Llama3-8B-Instruct on math (26.73 vs 20.05) and 1.08 over CoT for Llama3.1-8B-Instruct (36.97 vs 35.89).
- On RewardBench970, ReMA improves Llama3.1-8B-Instruct from 69.48 to 83.71 (+14.23).
- For Qwen2.5-7B-Instruct, ReMA gains +13.33 on AIME24 (6.67 to 20.00), but on the judge benchmark ReMA trails MRP_RL on RewardBench970 (83.51 vs 86.49).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use separate meta-thinking and reasoning agents trained with reinforcement learning instead of VRP (CoT) | ReMA · Math benchmarks, average across benchmarks | accuracy: +6.68% on Llama-3-8B-Instruct; +1.08% on Llama-3.1-8B-Instruct; +2.85% on Qwen2.5-7B-Instruct | Helped |
| Use separate meta-thinking and reasoning agents trained with reinforcement learning instead of VRP (CoT) | ReMA · AMC23 | accuracy: 2.50 -> 22.50; 27.50 -> 20.00; 47.50 -> 57.50 | Mixed |
| Use separate meta-thinking and reasoning agents trained with reinforcement learning instead of VRP (CoT) | ReMA · GSM8K | accuracy: 67.48 -> 79.38; 86.05 -> 87.26; 92.04 -> 90.60 | Mixed |
| Use separate meta-thinking and reasoning agents trained with reinforcement learning instead of VRP (CoT) | ReMA · RewardBench970 | accuracy: 69.48 -> 83.71; 78.56 -> 83.51 | Mixed |
| Use separate meta-thinking and reasoning agents trained with reinforcement learning instead of VRP (CoT) | ReMA · JudgeBench | accuracy: 51.29 -> 52.90; 58.39 -> 56.94 | Mixed |
| Use turn-level ratio instead of token-level ratio in multi-turn GRPO (ablation) | Multi-turn ReMA · 133-sample MATH training set | training reward / sample efficiency | Helped |
| Share agent parameters and update simultaneously instead of using separate weights (ablation) | Multi-turn ReMA · 133-sample MATH training set | convergence speed | Helped |
| Use meta-thinking plans for SFT before RL instead of CoT answers for SFT before RL (ablation) | Qwen2.5-Math-7B · AIME24 | accuracy | Helped |
| Scale meta-thinking agent from 1B to 8B (ablation) | Llama meta-thinking agent · MATH training | adaptive strategy use | Helped |
| Train multi-turn ReMA rather than use the single-turn setup | Qwen2.5-7B · Seven math test benchmarks | accuracy: about 5% | Helped |
| Use multi-turn rollout configuration sensitive to response length and turn count (ablation) | Multi-turn ReMA · Multi-turn math reasoning | training stability | Hurt |
Surprising findings
- Gains from RL on instruction-tuned models are smaller than on base models, and VRP_RL degrades Llama3.1-8B on several math benchmarks relative to CoT.
- Smaller 1B model collapses to the simplest EMPTY meta-action to avoid format penalties, while the 8B model adapts meta-thinking to problem difficulty.
Limitations
- Only evaluated on math and LLM-as-a-Judge benchmarks.
- Multi-turn training is highly sensitive to hyperparameters and can collapse into repetition or empty responses (the 'Echo Trap').
Experimental setup
Datasets: MATH RewardBench LIMO GSM8K AIME24 AMC23 GaoKao2023En Minerva Math OlympiadBench JudgeBench
Benchmarks: MATH500 GSM8K AIME24 AMC23 GaoKao2023En Minerva Math OlympiadBench RewardBench970 JudgeBench
Metrics: pass@1 (greedy decoding accuracy)
Baselines: VRP (CoT) VRP_RL MRP_RL RL from SFT RL from Base
Builds on: GRPO REINFORCE++ chain-of-thought TRPO
Closest papers by meaning
- Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.91)
- Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learning (AAAI 2026 · similarity 0.91)
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.90)
- DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs (AAAI 2026 · similarity 0.89)
- Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation (ICLR 2026 · similarity 0.89)
- Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.