Research map / Large language models / Agents, code and math
Episodic Memories Generation and Evaluation Benchmark for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a synthetic, contamination-free episodic memory benchmark for LLMs (narrative books with time/space/entity/content events and cue-based QA) and evaluates GPT-4o, Claude, Llama 3.1 and o1-mini under in-context, RAG and fine-tuning memory.
Problem
LLMs lack a robust episodic memory mechanism, and existing long-context benchmarks (needle-in-a-haystack, long-context QA, bAbI) do not test tracking of time, space, entity states and cue-based recall of specific events.
Approach
Model episodic events as tuples (time, space, entities, content) and generate controlled synthetic narrative books where ground-truth events are placed in chapters, then query them with all combinations of cues and trace types, including unanswerable cues to test confabulation.
Key results
- GPT-4o in-context and Claude 3.5 Sonnet RAG achieve the highest average ranks on the long book with no statistically significant difference between them.
- o1-mini reaches F1 0.97 on zero-event (unanswerable) questions in-context but drops to 0.05 on single-event questions, while fine-tuned GPT-4o-mini reaches 0.83 on single-event questions but 0.00 on zero-event questions.
- Latest-state recall is at most 36% for any model and recall of all events at most 18% on multi-event questions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RAG instead of in-context memory | LLMs on the long-book episodic-memory benchmark · Long book, overall questions | F1 average rank | Helped |
| Fine-tune on single-event question-answer pairs instead of using in-context memory | GPT-4o-mini · Long book, 1 matching event | F1: 0.54±0.46 -> 0.83±0.35 | Helped |
| Fine-tune on single-event question-answer pairs instead of using in-context memory | GPT-4o-mini · Long book, 2 matching events | F1: 0.44±0.36 -> 0.37±0.32 | Hurt |
| Fine-tune on single-event question-answer pairs instead of using in-context memory | GPT-4o-mini · Long book, 3–5 matching events | F1: 0.47±0.27 -> 0.28±0.21 | Hurt |
| Fine-tune on single-event question-answer pairs instead of using in-context memory | GPT-4o-mini · Long book, 6+ matching events | F1: 0.50±0.17 -> 0.19±0.07 | Hurt |
| Use RAG instead of in-context memory | GPT-4o-mini · Long book, 0 matching events | F1: 0.51±0.50 -> 0.63±0.49 | Helped |
| Increase the number of events matching the cue | LLMs on the episodic-memory benchmark · Long book, questions with two or more matching events | F1 | Hurt |
| Use time cues rather than context cues | LLMs on the episodic-memory benchmark · Long book, recall questions | F1 | Hurt |
| Track latest entity states across multiple events | LLMs on the episodic-memory benchmark · Long book, at least 2 events | Exact latest-state match: ≤36% | Hurt |
| Retrieve all events rather than only the latest state | LLMs on the episodic-memory benchmark · Long book, at least 2 events | Exact all-state match: ≤18% | Hurt |
Surprising findings
- o1-mini scores near-perfectly on unanswerable questions yet almost fails single-event recall in-context.
- Naive fine-tuning overfits to single-event facts and fails to generalize to multi-event questions, with the fine-tuned model scoring 0.00 on zero-event questions.
Limitations
- Relies on explicit temporal markers rather than implicit or relative time expressions.
- Independent chapter generation lacks causal and interconnected event structure.
- Limited domain scope: fictional human-like protagonists in New York-style settings.
Experimental setup
Datasets: synthetic episodic memory books generated with Claude 3.5 Sonnet
Benchmarks: Episodic memory benchmark (long book, 686 questions) Episodic memory benchmark (short book, 456 questions)
Metrics: F1-score Kendall's tau exact match fraction
Baselines: GPT-4o GPT-4o-mini Claude 3 Haiku Claude 3.5 Sonnet Llama 3.1 405B o1-mini
Builds on: needle-in-a-haystack bAbI LSQ (Latent Structure Queries)
Closest papers by meaning
- Human-inspired Episodic Memory for Infinite Context LLMs (ICLR 2025 · similarity 0.94)
- REMem: Reasoning with Episodic Memory in Language Agent (ICLR 2026 · similarity 0.92)
- WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.89)
- Spatially-Aware Transformers for Embodied Agents (ICLR 2024 · similarity 0.88)
- Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.88)
- EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.