Research map / Multi-agent systems and games / LLM agents

MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains LLM agents with end-to-end RL (PPO) to keep a single consolidated internal state per turn, enabling constant-memory multi-turn QA and web navigation, with multi-objective QA compositions for training.

Problem

Long-horizon multi-turn agents typically append all past thoughts, actions and observations to context, causing unbounded memory growth, higher inference cost, and degraded reasoning on out-of-distribution lengths.

Approach

Train the agent with RL to write a compact internal state <IS> that blends memory consolidation and reasoning, pruning all earlier turn content from the context; a masked-trajectory and two-dimensional attention mask make policy-gradient updates valid under this dynamic context, and multi-objective QA is built by composing existing QA questions.

Key results

• On 16-objective QA, MEM1-7B achieves 3.5x performance improvement over Qwen2.5-14B-Instruct while reducing memory 3.7x (abstract); it uses 27.1% of peak tokens and 29.3% of inference time versus Qwen2.5-14B-Instruct.
• On WebShop, MEM1-WebShop reaches an average final reward of 70.87, above AgentLM-13B (70.80) and AgentLM-7B (63.60), with 0.81x10^3 peak tokens versus 2.24x10^3 for AgentLM-7B.
• On 8-objective QA, MEM1-QA gets EM 1.87 and F1 2.31, while Search-R1 and DeepResearcher collapse (EM 0.064 and 0.73).

Surprising findings

• Adding a format reward speeds convergence but lowers final performance (EM 0.466 vs 0.709 on 2-objective QA).
• SFT on curated trajectories underperforms RL and collapses beyond six objectives.

Limitations

• Assumes environments with well-defined, verifiable rewards; open-ended tasks not addressed.
• Position-id approximation in the 2D attention mask means the original attention is not fully recovered.
• Reward-hacking/shortcut behavior observed during training (reduced searches early in RL).

Experimental setup

Datasets: HotpotQA Natural Questions Wikipedia 2018 dump WebShop 2wiki Bamboogle

Benchmarks: Multi-objective HotpotQA/NQ (2 to 16 objectives) Wiki RAG single-objective QA Online Web-QA WebShop

Metrics: exact match F1 peak token usage dependency length inference time average final reward

Baselines: Qwen2.5-14B-Instruct Qwen2.5-7B-Instruct Search-R1 DeepResearcher A-MEM Agent-FLAN Agent-R AgentLM GPT-4o SFT on GPT-4o trajectories

Builds on: PPO Search-R1 ReAct veRL HotpotQA

Closest papers by meaning

• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.92)
• Mem-T: Densifying Rewards for Long-Horizon Memory Agents (ICML 2026 · similarity 0.91)
• MemGen: Weaving Generative Latent Memory for Self-Evolving Agents (ICLR 2026 · similarity 0.91)
• Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents (ICLR 2026 · similarity 0.91)
• From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory (ICML 2026 · similarity 0.90)
• MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents (NeurIPS 2026 · similarity 0.90)