Research map / Large language models / Language, safety and interpretability
Human-inspired Episodic Memory for Infinite Context LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EM-LLM, a training-free method that segments the KV cache into surprise-based, graph-refined episodic events and retrieves them via similarity and temporal contiguity buffers, enabling near-infinite context in pretrained LLMs.
Problem
Pretrained LLMs struggle to process extremely long contexts; softmax attention is costly and diluted, and existing retrieval methods (e.g., InfLLM fixed-size blocks) or RAG are limited in precision and efficiency.
Approach
Segment the token stream into episodic events using Bayesian-surprise thresholds, refine boundaries by optimizing graph modularity/conductance over attention-key similarity, and retrieve events per layer with a two-stage similarity plus temporal-contiguity buffer, all without fine-tuning.
Key results
- EM-LLM with LLaMA-3.1-8B achieves 51.3 LongBench average vs 51.1 for InfLLM and outperforms NV-Embed-v2 RAG by about 30.5% on LongBench (47.24 vs 39.27 for LLaMA-3-8B).
- EM-LLM achieves 100% accuracy on Passkey retrieval across sequences up to 10.2M tokens.
- Surprise-based segmentation with refinement correlates with human-perceived event boundaries better than fixed segmentation (Fig. human comparison).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EM-LLM instead of InfLLM | Mistral-7B-Instruct-v0.2 · LongBench Avg. score | score: 41.90 -> 43.47 | Helped |
| use EM-LLM instead of InfLLM | LLaMA-3.1-8B-Instruct · ∞-Bench Avg. score | score: 64.00 -> 66.69 | Helped |
| use EM-LLM instead of InfLLM | LLaMA-3-8B-Instruct · ∞-Bench Avg. score | score: 50.31 -> 49.00 | Hurt |
| use EM-LLM SM instead of RAG with NV-Embed-v2 | LLaMA-3-8B-Instruct · LongBench Avg. score | score: 39.27 -> 47.24 | Helped |
| use EM-LLM S instead of full-context processing | LLaMA-3.1-8B · LongBench Avg. score | score: 39.30 -> 51.58 | Helped |
| add modularity boundary refinement to surprise-based segmentation (ablation) | EM-LLM · LongBench and ∞-Bench tasks | task score: best performance in 60% of tasks | Helped |
| add a contiguity buffer to surprise-based retrieval (ablation) | EM-LLM · LongBench and ∞-Bench tasks | task score: best performance in 44% of tasks | Mixed |
| use surprise-based boundaries instead of fixed-size segmentation (ablation) | event segmentation · PG-19 and human-annotated podcast scripts | similarity metrics | Helped |
| use surprise-based chunking for RAG instead of fixed-size chunking (ablation) | RAG · LongBench | score | Hurt |
Surprising findings
- Fixed-size InfLLM segmentation performs worse than random segmentation on cohesion/separation metrics.
- Surprise-based chunking for RAG (RAG-S) underperforms fixed-size chunking RAG.
Limitations
- Gains on individual tasks are often not statistically significant; benefits vary across tasks (e.g., MultiNews, Math.Find, En.MC sometimes worse than InfLLM).
- Performance on some tasks with Phi-3.5 benchmark-level significance p=0.23 for LongBench.
Experimental setup
Datasets: PG-19 human-annotated podcast transcripts
Benchmarks: LongBench ∞-Bench
Metrics: accuracy F1 ROUGE modularity conductance intra/inter-similarity ratio Wasserstein distance
Baselines: InfLLM RAG (NV-Embed-v2) RAG (all-mpnet-base-v2) full-context
Builds on: InfLLM RoPE attention sinks (StreamingLLM) k-NN attention retrieval
Closest papers by meaning
- Episodic Memories Generation and Evaluation Benchmark for Large Language Models (ICLR 2025 · similarity 0.94)
- REMem: Reasoning with Episodic Memory in Language Agent (ICLR 2026 · similarity 0.90)
- PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding (NeurIPS 2025 · similarity 0.88)
- From RAG to Memory: Non-Parametric Continual Learning for Large Language Models (ICML 2025 · similarity 0.87)
- Scaling and context steer LLMs along the same computational path as the human brain (NeurIPS 2025 · similarity 0.87)
- E-mem: Multi-Agent Based Episodic Context Reconstruction for LLM Agent Memory (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.