Research map / Multi-agent systems and games / LLM agents

Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RL-Focal, a two-stage multi-agent RL framework (Decider selects a small LLM ensemble, Fusion resolves conflicts) guided by a focal diversity metric, evaluated on five MCQ/math benchmarks against individual LLMs and ensemble baselines.

Problem

Static or fixed-size LLM ensembles lack adaptability to changing tasks and query distributions, and existing routing and ensemble methods are limited by the best model in the pool, high inference cost, or supervised training.

Approach

Two cooperating RL agents trained with MAPPO and a centralized critic: a Decider that picks a small subset of N LLMs per query using a focal-diversity-aware state and a size-penalized reward, and a Fusion agent that picks the final answer from the selected models' answer probabilities.

Key results

• RL-Focal reaches 65.00% on BBH versus the best single model Phi-4-14b at 59.94% (8.48% relative gain as reported in the table header).
• On MMLU and GPQA with the RLAE model pool, RL-Focal reaches 69.2 and 36.78, versus RLAE-MAPPO at 69.1 and 35.3, at about 2x to 9x lower cost.

Surprising findings

• Best-average models such as Phi-4-14B, Qwen2.5-72B and Llama-3-70B are often not selected by the Decider on BBH and MMLU subsets, favoring complementary models instead.
• A reported Fusion accuracy of 0.32% at discount factor 0.99 shows PPO fails to converge at high gamma.

Limitations

• Fusion and Decider are restricted to multiple-choice format; open-ended extension is only sketched.
• Relies on per-option answer probabilities from models, and GSM8K probabilities come from repeated sampling (10 passes).
• Results vary by ensemble and some baselines are taken from other reports, making comparisons less controlled.

Experimental setup

Datasets: MMLU BBH MUSR GPQA GSM8K Alpaca-cleaned BeaverTails TruthfulQA HuggingFace Open LLM Leaderboard logits

Benchmarks: MMLU BBH MUSR GPQA GSM8K

Metrics: accuracy cost (cents) inference count

Baselines: RouteLLM RLAE-MAPPO RLAE-PPO PairRanker GAC DEEPEN LLM-TOPLA LLM-Blender More Agents DyLAN Majority Voting Mixtral-8x7b

Builds on: MAPPO GAE RLAE RouteLLM

Closest papers by meaning

• Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.89)
• DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs (AAAI 2026 · similarity 0.89)
• HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.88)
• Foundations of Multi-Agent Learning in Dynamic Environments: Where Reinforcement Learning Meets Strategic Decision-Making (AAAI 2025 · similarity 0.88)
• Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity (ICLR 2026 · similarity 0.88)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.88)