Research map / Large language models / Reasoning and chain of thought

MEMAUDIT: An Exact Package-Oracle Evaluation Protocol for Budgeted Long-Term LLM Memory Writing

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MemAudit, an exact package-oracle protocol that scores LLM memory writers against a certified budgeted optimum over a finite candidate set of memory representations with concave-over-modular semantic coverage.

Problem

Existing LLM memory evaluations measure only end-to-end QA accuracy, which entangles memory writing with retrieval and reader reasoning, so write-time memory quality cannot be isolated.

Approach

Turn write-time memory selection into a finite knapsack-plus-partition-matroid optimization over experience-representation choices with a monotone submodular semantic coverage objective, certify the optimum via branch-and-bound and MILP, and score external systems (Mem0, A-Mem, Letta) under a union denominator.

Key results

• On exact-small packages, MemAudit-GVT stays near certified OPT across budgets B=2,4,8,16 while density-only covers no invalidation units at B=2,4,8.
• On Natural-87 at B=100, Estimated-GVT reaches 0.83 of package OPT, and Letta salience-pruned achieves union ratio 0.734 versus Mem0 salience-pruned 0.427.
• Branch-and-bound matches PuLP MILP objective on 1,200/1,200 audit rows with max diff 0.0.

Surprising findings

• Mem0's upper-pruned package ratio exceeds 1.0 because its exports add coverage absent from package candidates, so the union denominator is needed.
• A-Mem full-store view scores zero at B=100 because no full serialized note fits the budget.

Limitations

• Scores are package-conditional and depend on the candidate table, cost model, and evidence schema.
• Natural-87 is small, support-sliced, and relies on model-adjudicated labels with moderate human agreement for Sonnet.

Experimental setup

Datasets: LongMemEval-S Natural-87 (support-slice package from LongMemEval-style data) MemSim-Small synthetic packages

Benchmarks: MemAudit-Small exact packages MemAudit validity-heavy stress packages Natural-87 LongMemEval-S

Metrics: package ratio to OPT union ratio upper-pruned bound Cohen's kappa invalidation coverage token-F1

Baselines: GVT density-only recency raw fact-only summary-only Mem0 A-Mem Letta

Builds on: MemSim/MemDaily LongMemEval Mem0 A-Mem Letta

Closest papers by meaning

• MEME: Multi-Entity & Evolving Memory Evaluation (NeurIPS 2026 · similarity 0.88)
• Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (NeurIPS 2026 · similarity 0.87)
• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.87)
• A-MemGuard: A Proactive Defense Framework For LLM-Based Agent Memory (ICML 2026 · similarity 0.87)
• Minerva: A Programmable Memory Test Benchmark for Language Models (ICML 2025 · similarity 0.86)
• Mem-T: Densifying Rewards for Long-Horizon Memory Agents (ICML 2026 · similarity 0.86)