Research map / Multi-agent systems and games / LLM agents

From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

ICML 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a plug-in memory model (MemoPilot) with multi-turn GRPO using turn-wise one-step rewards to update textual memory that guides a frozen LLM player across sequential Rock-Paper-Scissors and Limit Texas Hold'em games.

Problem

Test-time learning for LLM agents over sequential interactions: hand-designed prompt-based memory update rules fail to consistently improve performance across repeated games, and memory updates are not optimized for downstream objectives.

Approach

Treat memory updating as a multi-turn MDP and train the memory generator end-to-end with multi-turn GRPO, using per-turn advantages computed from one-step next-game rewards (rather than cumulative returns) and a structured 3-tier memory (identification, maintenance, guidance), while the player stays frozen.

Key results

• MemoPilot reaches RPS@5 3.28 and LHE@5 2.03 with Qwen2.5-14B as frozen player, versus 0.43 and -1.36 for No Memory and 0.81/-0.87 for best prior method (ReasoningBank with Qwen3 player).
• MemoPilot ranks first in Elo on both games (1590 on RPS, 1762 on LHE).
• A memory model trained with Qwen2.5-14B transfers to frozen Qwen3-235B-A22B, achieving RPS@5 3.27 and LHE@5 1.31.

Surprising findings

• Full History degrades performance on RPS (0.02 vs 0.43 for No Memory), suggesting longer raw histories add noise.
• Cumulative (multi-step) reward for memory training yields only 0.61 on LHE@5, far below one-step reward (2.03), due to future environment variance.

Limitations

• Degrades under non-stationary or adaptive opponents, e.g., LHE@5 falls from 2.03 to 1.21 when opponent switches every 2 games.
• Requires informative experience and dense enough reward; sparse rewards limit memory learning.
• Fixed 512-token memory budget; very long trajectories need chunking or summarization.

Experimental setup

Datasets: TextArena RLCard StreamBench

Benchmarks: Rock-Paper-Scissors (RPS@5) Limit Texas Hold'em (LHE@5) StreamBench (CoSQL, DS-1000)

Metrics: mean@64 per-game score Elo rating pass@4 accuracy

Baselines: No Memory Full History Human-Written Counter-Strategy Reflexion ExpeL MemoryBank AWM ReasoningBank Memory w/ DeepSeek-V3.2 Memory w/ Gemini-3.0-Flash

Builds on: GRPO Reflexion ExpeL Dynamic Cheatsheet ReasoningBank

Closest papers by meaning

• MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games (ICML 2026 · similarity 0.92)
• Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization (ICLR 2026 · similarity 0.91)
• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.91)
• MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (ICLR 2026 · similarity 0.90)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.89)
• Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.89)