Research map / Large language models / Reasoning and chain of thought

LightMem: Lightweight and Efficient Memory-Augmented Generation

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LightMem, a three-stage Atkinson-Shiffrin-inspired memory system (pre-compression sensory memory, topic-aware STM, sleep-time offline-updated LTM) that improves QA accuracy and cuts token/API costs on LongMemEval and LoCoMo.

Problem

Existing LLM memory systems incur heavy token, API-call and latency overhead from processing raw dialogue with LLMs and from updating memory during inference, while ignoring efficiency.

Approach

Organizes memory into three stages inspired by human memory: LLMLingua-2 pre-compression of raw turns, hybrid attention+similarity topic segmentation before summarization, and a long-term store that appends entries at test time (soft update) and performs parallel, timestamp-constrained updates offline ("sleep-time").

Key results

• On LongMemEval-S with GPT-4o-mini, LightMem (r=0.7, th=512) reaches 68.64% ACC vs 61.00% for NaiveRAG and 62.60% for A-MEM, with 28.25k total tokens vs 1,605.81k for A-MEM (online-only figures are lower still).
• On LongMemEval-S with Qwen3-30B, LightMem (r=0.6, th=768) reaches 70.20% ACC vs 65.20% for A-MEM, and total token use drops to about 25-32k vs 1,864.93k for A-MEM.
• On LoCoMo, LightMem improves ACC by up to 29.29% over baselines on Qwen and up to 20.92x total token efficiency on GPT.

Surprising findings

• Removing the topic segmentation module slightly improves efficiency but drops accuracy by 6.3% (GPT) and 5.4% (Qwen).
• Optimal compression ratio for accuracy depends on buffer threshold; larger buffers favor less compression.

Limitations

• Relies on an additional compression model (LLMLingua-2) and attention-based segmentation, adding pipeline complexity.
• Accuracy is non-monotonic in buffer threshold and compression ratio, requiring tuning.
• Evaluated only on two conversational benchmarks with GPT-4o-mini judge; LongMemEval-S subset of 5 corrupted samples discarded.

Experimental setup

Datasets: LongMemEval-S LoCoMo

Benchmarks: LongMemEval-S LoCoMo

Metrics: accuracy total tokens API calls runtime

Baselines: FullText NaiveRAG LangMem A-MEM MemoryOS Mem0

Builds on: LLMLingua-2 Atkinson-Shiffrin memory model A-MEM Mem0 MemoryOS

Closest papers by meaning

• M+: Extending MemoryLLM with Scalable Long-Term Memory (ICML 2025 · similarity 0.89)
• SimpleMem: Efficient Lifelong Memory for LLM Agents (ICML 2026 · similarity 0.88)
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025 · similarity 0.88)
• A-Mem: Agentic Memory for LLM Agents (NeurIPS 2025 · similarity 0.87)
• Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.87)
• PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents (ICML 2026 · similarity 0.86)