Research map / Multi-agent systems and games / LLM agents

MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MemRL, a non-parametric method that keeps the LLM frozen and learns a value-based retrieval policy over an Intent-Experience-Utility episodic memory via Monte Carlo Q-value updates from environment feedback.

Problem

Agents need to keep improving after deployment without fine-tuning (which is costly and causes catastrophic forgetting), but existing memory-based methods retrieve by semantic similarity and often pull in noisy, low-utility experiences.

Approach

Store experiences as Intent-Experience-Utility triplets and treat memory retrieval as an RL decision: a two-phase retrieval filters by similarity then ranks by a z-score-normalized mix of similarity and learned Q-value, with Q updated by a Monte Carlo rule from task reward while the LLM stays frozen.

Key results

• Runtime learning average last-epoch/CSR of 0.772/0.798 vs. 0.736/0.760 for strongest baseline MemP across the five settings.
• Transfer learning average success rate 0.794 vs. 0.766 for MemP across BigCodeBench, Lifelong Agent Bench (OS/DB) and ALFWorld.
• Learned Q-values correlate with empirical success (Pearson r=0.861), rising from 21.5% to 88.1% success across bins.

Surprising findings

• On HLE, the single-task reflection ablation (0.610) ties or slightly beats full cross-task MemRL (0.606).
• High-Q memory bins still contain roughly 12% failure memories, which the authors interpret as useful near-miss guidance.

Limitations

• Step-wise updates may be high-variance for long-horizon trajectories; credit assignment across multiple referenced experiences is ambiguous.
• Gains depend on task similarity; with low similarity performance drifts toward reflection-like behavior.
• Backbones are selected per benchmark, so cross-benchmark comparisons use different models.

Experimental setup

Datasets: BigCodeBench ALFWorld Lifelong Agent Bench HLE

Benchmarks: BigCodeBench ALFWorld Lifelong Agent Bench HLE

Metrics: success rate cumulative success rate forgetting rate

Baselines: No Memory Pass@10 RAG Self-RAG Mem0 MemP Reflexion

Builds on: Reflexion Monte Carlo value estimation Generalized EM Mem0 MemP

Closest papers by meaning

• MemEvolve: Meta-Evolution of Agent Memory Systems (ICML 2026 · similarity 0.92)
• MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents (NeurIPS 2026 · similarity 0.92)
• MemGen: Weaving Generative Latent Memory for Self-Evolving Agents (ICLR 2026 · similarity 0.91)
• MAGE: Multi-Agent Self-Evolution with Co-Evolutionary Knowledge Graphs (NeurIPS 2026 · similarity 0.90)
• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.89)
• MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (ICLR 2026 · similarity 0.89)