Research map / Large language models / Agents, code and math
Minerva: A Programmable Memory Test Benchmark for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Minerva, a programmable benchmark that auto-generates atomic and composite memory tests (search, recall/edit, match/compare, state tracking) to diagnose how LLMs use their context, and evaluates nine models at 4k tokens.
Problem
Static, manually crafted data benchmarks for long-context use are costly, overfit-prone, and fail to pinpoint which specific memory capability a model lacks; existing automated tests mostly probe only basic search (needle-in-a-haystack, passkey, key-value).
Approach
A programmable framework of parametric test generators covering atomic memory capabilities (beyond search: recall/edit, match/compare, spot differences, set/list computation, stateful processing) plus composite tests (data blocks, multi-agent state tracking), so each test isolates a capability and difficulty is tunable.
Key results
- GPT-4o scored 1.00 on word string search but only 0.82 on subsequence search and 0.00-level collapse for smaller models on quantity state tracking (e.g., Mistral-7B, Phi-3, LLaMA-3.1-8B all 0.0), while GPT-4o reached 1.0.
- On composite tests, the best model GPT-4o scored 0.37 on Processing Data Blocks and 0.38 on Theory of Mind, with all models performing substantially lower than on atomic tests.
- Performance on functional updates for GPT-4o dropped from 1.00 at 500 tokens to 0.59 at 8K tokens; gpt-4o-mini dropped from 0.69 to 0.10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace meaningful context words with random gibberish tokens (ablation) | GPT-4o snapshot and string search · Snapshot and string search | accuracy | No clear effect |
| Return the last rather than the first word in each list (ablation) | Iterate task · Compute on Sets and Lists | ROUGE-L | Hurt |
| Increase functional-update context length from 500 to 8K tokens (ablation) | GPT-4o · Functional updates | ROUGE-L: 1.00 -> 0.59 | Hurt |
| Increase counting context length from 1K to 16K tokens (ablation) | GPT-4-turbo · Counting | exact match accuracy: 0.52 -> 0.28 | Hurt |
| Use minor alternate prompt phrasing (ablation) | GPT-4o · String search | exact match accuracy: 1.00 -> 1.00 | No clear effect |
| Use minor alternate prompt phrasing (ablation) | GPT-4o · Group association | exact match accuracy: 0.65 -> 0.63 | No clear effect |
| Use minor alternate prompt phrasing (ablation) | GPT-4o · Replace all | ROUGE-L: 0.99 -> 0.98 | No clear effect |
| Use alternate prompt phrasing (ablation) | Phi-3-small · Check association | exact match accuracy: 0.70 -> 0.60 | Mixed |
| Combine multiple atomic capabilities in a composite task | GPT-4o · Memory processing versus composite tests | performance: 0.37 processing data blocks; 0.38 theory of mind | Hurt |
| Use alternating repeated groups rather than standard group structures (ablation) | Compute on Sets and Lists · Group association | accuracy: 0.75 -> 0.68 | Hurt |
Surprising findings
- Models perform near-perfectly on search tasks yet collapse on stateful and composite tasks, showing needle-in-a-haystack performance does not imply effective context processing.
- An 8B LLaMA model outperformed GPT-4 on the Identify the Odd Group task, suggesting model size is not the sole determinant.
Limitations
- Evaluation mostly at short 4k context and a fixed snapshot of 1110 samples, with small per-task sample sizes.
- Tests are synthetic word lists and may not reflect real-world memory use such as emails or calendars.
- Ablation/prompt variation studies are limited to few models and tasks.
Experimental setup
Datasets: Minerva programmatically generated test cases
Benchmarks: Minerva
Metrics: exact match ROUGE-L ROUGE-L recall Jaccard similarity
Baselines: GPT-4-turbo GPT-4o GPT-4o-mini Cohere-command-rplus Mistral-7B-instruct-v0.2 Phi-3-small Phi-3-medium Gemma-2-9b LLaMA-3.1-8B-instruct
Builds on: needle-in-a-haystack RULER NeedleBench
Closest papers by meaning
- Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.91)
- EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.90)
- AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.89)
- AUTOMEM: Automated Learning of Memory as a Cognitive Skill (NeurIPS 2026 · similarity 0.89)
- AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks (NeurIPS 2025 · similarity 0.89)
- AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.