Research map / Large language models / Retrieval and knowledge
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LongMemEval, a 500-question benchmark for five long-term memory abilities of chat assistants, and studies memory design choices (value decomposition, fact-augmented key expansion, time-aware query expansion, chain-of-note JSON reading) across indexing, retrieval and reading.
Problem
Existing long-term dialogue memory benchmarks use short, human-human or non-task-oriented histories and cover few memory abilities (multi-session reasoning, knowledge updates, temporal reasoning, assistant-side recall, abstention), so chat assistant memory is poorly evaluated.
Approach
Builds a benchmark of 500 human-curated questions embedded in freely extensible, timestamped user-assistant chat histories (self-chat evidence sessions mixed into ShareGPT/UltraChat haystacks), and decomposes memory systems into indexing, retrieval and reading stages with four control points (value, key, query, reading strategy) to identify effective designs.
Key results
- Commercial assistants ChatGPT and Coze with GPT-4o show 37% and 64% performance drops versus offline reading on a short-history subset, and long-context LLMs drop 30%-60% on LongMemEval_S versus oracle retrieval.
- Fact-augmented key expansion (K = V + fact) with rounds as values reaches Recall@5 of 0.644 and GPT-4o Top-5 QA accuracy of 0.657 on LongMemEval_M, versus 0.582 and 0.615 for K = V.
- Time-aware query expansion improves temporal-reasoning recall by 11.3% with round values and 6.8% with session values when GPT-4o extracts time ranges.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Expand value keys with extracted user facts | Long-term memory RAG with Stella V5 retrieval · LongMemEval-M | Recall@5: 0.706 -> 0.732 | Helped |
| Expand value keys with extracted user facts | Long-term memory RAG with GPT-4o reader · LongMemEval-M | Top-5 QA: 0.670 -> 0.714 | Helped |
| Replace the original value key with extracted facts alone (ablation) | Long-term memory RAG · LongMemEval-M, Value = Round | Recall@5: 0.582 -> 0.530 | Hurt |
| Add time-aware query expansion using GPT-4o | Long-term memory retrieval with session values and K = V · LongMemEval-M temporal reasoning | Recall@5: 0.639 -> 0.654 | Helped |
| Use Llama 3.1 8B Instruct instead of GPT-4o for time-aware query expansion (ablation) | Long-term memory retrieval with session values and K = V · LongMemEval-M temporal reasoning | Recall@5: 0.639 -> 0.624 | Hurt |
| Decompose sessions into rounds (ablation) | RAG with GPT-4o reader · LongMemEval-M | QA performance | Helped |
| Replace sessions or rounds with extracted summaries or facts (ablation) | Long-term memory RAG · LongMemEval-M | QA performance | Hurt |
| Replace sessions or rounds with fact decomposition (ablation) | Long-term memory RAG · LongMemEval-M multi-session reasoning | accuracy | Helped |
| Apply Chain-of-Note with JSON format instead of a suboptimal reading strategy (ablation) | Reader LLM under oracle retrieval · LongMemEval | QA accuracy: 10-point absolute performance drop | Helped |
| Use Chain-of-Note with JSON format rather than natural-language format (ablation) | Reader LLM · LongMemEval oracle retrieval | QA accuracy: as much as 10 absolute points | Helped |
Surprising findings
- Using condensed forms (summaries, keyphrases, facts) alone as keys does not improve memory recall over the raw value, despite more focused semantics.
- Even with perfect (oracle) retrieval, a suboptimal reading strategy causes up to a 10-point accuracy drop for GPT-4o.
Limitations
- Commercial system evaluation was a small manual study on 97 questions with short 3-6 session histories, and the abstention and some temporal questions were excluded.
- Benchmark relies on LLM-generated and LLM-simulated sessions, with about 70% human-edited, which may not fully reflect real user chats.
- Time-aware query expansion depends on a strong LLM; Llama 3.1 8B struggles to produce accurate time ranges.
Experimental setup
Datasets: ShareGPT UltraChat LongMemEval (self-chat evidence sessions)
Benchmarks: LongMemEval_S LongMemEval_M
Metrics: accuracy Recall@k NDCG@k
Baselines: ChatGPT Coze In-context RAG MemoryBank LD-Agent RAPTOR MemWalker HippoRAG
Builds on: Chain-of-Note needle-in-a-haystack test HippoRAG RAPTOR MemoryBank
Closest papers by meaning
- AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations (ICLR 2026 · similarity 0.93)
- MemoryBank: Enhancing Large Language Models with Long-Term Memory (AAAI 2024 · similarity 0.91)
- Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.90)
- Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction (AAAI 2026 · similarity 0.89)
- SeCom: On Memory Construction and Retrieval for Personalized Conversational Agents (ICLR 2025 · similarity 0.89)
- PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.