Research map / Large language models / Reasoning and chain of thought

MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancement

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MEML-GRPO, which fine-tunes a base LLM on answers from multiple expert system prompts, then runs GRPO per expert with a KL-style mutual-learning term from best to worst expert and an SFT buffer for hard questions, evaluated on GSM8K, MathQA and StrategyQA with Qwen2.5-1.5B-Math and Llama3.2-1B-Instruct.

Problem

Standard RLVR suffers from reward sparsity: when all sampled responses for hard questions are incorrect, GRPO gets no learning signal, and on-policy training struggles to go beyond the base model's existing reasoning.

Approach

Use diverse expert system prompts, each conditioned on responses from heterogeneous external models (DeepSeek-r1, Doubao-1.5-thinking, ground truth), to widen the set of sampled correct solutions; add an inter-expert log-likelihood regularizer that pushes the worst expert toward the best expert's correct outputs, and an SFT buffer of ground-truth examples for questions all experts fail.

Key results

• MEML-GRPO on Qwen2.5-1.5B-Math reaches 77.1 average accuracy (Expert1) vs 73.7 for the best MoE-SFT-Dr.GRPO baseline in the main table.
• On Llama3.2-1B-Instruct, MEML-GRPO (Expert0) averages 61.1 vs 55.7 for the best single-expert Dr.GRPO baseline.
• Majority voting over MoE-SFT-GRPO gives 73.6 average, while MEML-GRPO experts match or exceed majority voting at single-model inference cost (MEML-GRPO MV 76.9 vs best expert 77.1).

Surprising findings

• Single-expert SFT with the wrong-reasoning-style expert (DeepSeek-r1 prompt on Qwen) sometimes beats the ground-truth expert, and the MoE-SFT alone does not consistently beat single-expert SFT.

Limitations

• Experiments limited to two small models (1.5B and 1B) and three datasets.
• Inconsistent numbers across tables and text (e.g., ablation full config 77.3 vs main table 77.1; text says Qwen average 79.3 on GSM8K, table shows 80.1) suggest reporting problems.
• Mutual-learning loss is described only approximately as a KL term and no code is released.

Experimental setup

Datasets: GSM8K MathQA StrategyQA

Benchmarks: GSM8K MathQA StrategyQA

Metrics: exact-match accuracy average accuracy

Baselines: Expert0-SFT-GRPO Expert0-SFT-Dr.GRPO MoE-SFT-GRPO MoE-SFT-Dr.GRPO MoE-SFT majority voting

Builds on: GRPO Dr.GRPO RLVR

Closest papers by meaning

• Multi-Task GRPO: Reliable LLM Reasoning Across Tasks (ICML 2026 · similarity 0.88)
• R1-ShareVL: Incentivizing Reasoning Capabilities of Multimodal Large Language Models via Share-GRPO (NeurIPS 2025 · similarity 0.88)
• Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration (ICML 2026 · similarity 0.87)
• Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs (ICLR 2026 · similarity 0.87)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.87)
• Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs (ICLR 2026 · similarity 0.87)