Research map / Multi-agent systems and games / LLM agents
AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AMemGym, an interactive, schema-grounded environment with LLM-simulated users for on-policy evaluation and self-evolution of assistant memory, showing off-policy benchmarks can mis-rank memory systems.
Problem
Existing conversational memory benchmarks rely on static, off-policy dialogue contexts, which fail to reflect an assistant's own interactive behavior and are costly to curate, biasing evaluation and optimization of memory.
Approach
Reverse-engineer evaluation from structured data (user profiles, a global state schema, state evolution trajectories, and state-variant-specific answers verified by reflection), then run an LLM user simulator that exposes latent states through role-play so assistants are evaluated on-policy; memory is decomposed into write, read, and utilization failures and normalized into a memory score.
Key results
- Off-policy evaluation changes memory-system rankings; e.g., AWE-(2,4,30) scores .291 on-policy vs .253 off-policy and drops 3 ranks.
- All evaluated LLMs reach upper-bound scores above 0.8 on short contexts, but most fall below 50% of their upper bounds in long interactions.
- Complete feedback self-evolution raises the in-context agent's memory score from .172 to .197 and lowers write failures from .293 to .263.
Surprising findings
- Off-policy evaluation can reverse rankings of memory implementations, so static benchmarks may mislead configuration choices.
- Some LLMs perform no better than random guessing in later periods of long interactions.
Limitations
- Benchmark uses only 20 user profiles with 200 multiple-choice questions, a small evaluation set.
- Relies on gpt-4.1 for data generation, user simulation, and answer verification, so results depend on that model's behavior.
Experimental setup
Datasets: Nemotron-Personas
Benchmarks: AMemGym
Metrics: accuracy normalized memory score write failure rate read failure rate utilization failure rate
Baselines: LoCoMo LongMemEval Mem0-G Nemori A-Mem standard RAG native long-context LLM
Builds on: mem0 CollabLLM Nemotron-Personas
Closest papers by meaning
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025 · similarity 0.93)
- Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-Horizon Agents (NeurIPS 2026 · similarity 0.92)
- Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction (AAAI 2026 · similarity 0.91)
- Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.90)
- Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.90)
- AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.