Research map / Multi-agent systems and games / LLM agents

AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AMA-Bench, a benchmark for long-horizon agent memory built from real agent trajectories (six domains, 2,496 expert QA) plus programmatic synthetic trajectories (TextWorld, BabyAI), and proposes AMA-Agent, which combines a causality graph memory with tool-augmented retrieval.

Problem

Existing memory benchmarks are dialogue-centric and natural-language focused, while LLM agents need memory over machine-generated, causally grounded agent-environment trajectories with dense objective information.

Approach

Construct a causality graph from adjacent observation-action-observation turns (LLM-based semantic abstraction) and combine embedding search with graph-node traversal and keyword/script tools for retrieval, rather than lossy compression or similarity-only retrieval.

Key results

• AMA-Agent reaches 0.5722 average accuracy on the real-world subset with Qwen3-32B, versus 0.4610 for the strongest memory baseline EMem (about 11 points higher).
• GPT 5.2 long-context achieves 0.7226 average accuracy on the real-world subset, the highest reported.
• Ablation: removing the causality graph drops average accuracy from 0.57 to 0.43; removing tool-augmented retrieval drops it to 0.44.

Surprising findings

• Many existing memory systems underperform a simple long-context baseline on agent trajectories, despite outperforming it on dialogue benchmarks.
• Scaling the backbone from 8B to 32B gives only marginal gains (avg. about 0.038), while memory architecture drives most variance.

Limitations

• Focuses on in-episode memory; cross-task or lifelong memory is not studied.
• Offline per-query QA protocol does not capture sequential end-to-end agent execution, though a small E2E check is included on TextWorld and Spider2.

Experimental setup

Datasets: ALFWorld BALROG LMGame-Bench WebArena SWE-bench Verified GAIA Spider 2.0 TextWorld BabyAI

Benchmarks: AMA-Bench real-world subset AMA-Bench synthetic subset LoCoMo

Metrics: accuracy F1

Baselines: Long-context LLMs (GPT 5.2, GPT-5-mini, Claude Haiku 3.5, Gemini 2.5 Flash, Qwen BM25 Qwen3-Embedding-4B GraphRAG HippoRAG2 MemAgent MEM1 A-Mem Mem0 MemoRAG MemGPT MemoryBank

Builds on: HippoRAG2 GraphRAG MemGPT MemoRAG ReAct

Closest papers by meaning

• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.94)
• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.94)
• A-Mem: Agentic Memory for LLM Agents (NeurIPS 2025 · similarity 0.93)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.92)
• Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks (ICML 2026 · similarity 0.92)
• AdaMEM: Test-Time Adaptive Memory for Language Agents (ICML 2026 · similarity 0.92)