Research map / Multi-agent systems and games / LLM agents

Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MMPO, which adds a self-supervised Belief Entropy reward (entropy of an LLM's answer to an anchor question about memory) to outcome RL for training recursive memory summarization agents, reporting gains on RULER-HotpotQA, MEM1 multi-objective QA and WebShop.

Problem

Outcome-based RL for memory-augmented LLM agents gives sparse credit, failing to localize where intermediate recursive summaries degrade and accumulate noise, which causes belief deviation and long-horizon reasoning collapse.

Approach

Measure intermediate memory quality via the token-level predictive entropy of the model's response to a task-state anchor question conditioned on the current memory ("Belief Entropy"), and convert it into a sigmoid-normalized dense per-turn reward added to the terminal outcome reward, with turn-level group-relative advantages optimized by PPO-style clipping.

Key results

• MMPO-7B improves average accuracy over RL-MemAgent-7B by about +3.14% from 224K to 3.5M context on RULER-HotpotQA, with a +5.47% gain at 896K.
• On MEM1 WebShop, MMPO reaches 77.25 average reward versus 70.87 for MEM1-WebShop.
• Selecting the lowest-entropy trajectory among 5 candidates improves accuracy over Vanilla+Memory without training (Finding 3); entropy reduction correlates with accuracy at Pearson r=-0.684.

Surprising findings

• A direct-answer probe (rewarding low answer entropy) underperforms the outcome-only baseline (77.71 vs 78.91 average), suggesting rewarding answer confidence encourages premature certainty.

Limitations

• Belief Entropy is a proxy for summary-induced belief uncertainty rather than a direct measure of latent task-state uncertainty, and depends on anchor question design.
• Results on MEM1 multi-objective QA are reported with EM/F1 values that are very small for several settings, and the paper gives limited detail on training data size and compute.

Experimental setup

Datasets: HotpotQA RULER-HotpotQA MEM1 multi-objective QA WebShop

Benchmarks: RULER-HotpotQA MEM1 multi-objective QA WebShop

Metrics: accuracy EM F1 average reward

Baselines: MemAgent (RL-MemAgent) MEM1-QA MEM1-WebShop QwenLong-L1-32B DeepResearcher Search-R1 A-MEM

Builds on: MemAgent MEM1 GRPO RLVR

Closest papers by meaning

• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.90)
• MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (ICLR 2026 · similarity 0.90)
• MemGen: Weaving Generative Latent Memory for Self-Evolving Agents (ICLR 2026 · similarity 0.89)
• Mem-T: Densifying Rewards for Long-Horizon Memory Agents (ICML 2026 · similarity 0.89)
• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.89)
• AUTOMEM: Automated Learning of Memory as a Cognitive Skill (NeurIPS 2026 · similarity 0.89)