Research map / Large language models / Reasoning and chain of thought

Neural Garbage Collection: Learning to Forget while Learning to Reason

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Neural Garbage Collection (NGC), which trains a reasoning LM end-to-end with Dr. GRPO to sample KV-cache eviction decisions (Gumbel top-k over attention-derived block scores) alongside tokens, using only outcome reward.

Problem

Chain-of-thought reasoning grows the KV cache without bound, and existing eviction methods rely on hand-designed heuristics or proxy objectives rather than task reward.

Approach

Treat KV-cache eviction as a discrete action sampled from the LM (Gumbel top-k over attention-based block scores, parameter-free, phi=theta) and optimize it jointly with reasoning tokens via a single outcome-based RL reward, with replay attention masks to get correct off-policy log-probs.

Key results

• On Countdown at 50% eviction per round (2.4x peak KV reduction), NGC reaches 49.6% accuracy vs 21.2% for the next-best baseline.
• Token log-probs only ablation reaches 35.7% and targeted KV dropout 2.5% on Countdown versus NGC's 49.6%.
• On AMC/AIME at 3-5x cache reduction, NGC consistently outperforms SnapKV, KeyDiff, KNorm and StreamingLLM, with heuristics sometimes degrading to near-zero accuracy.

Surprising findings

• SnapKV fails catastrophically on Countdown but performs reasonably on math tasks, showing heuristic eviction rankings are task-dependent.
• Targeted KV dropout (ignoring off-policyness from evictions) collapses after around 100 training steps with exploding gradient norms.

Limitations

• Experiments limited to a single 1.5B model (DeepSeek-R1-Distill-Qwen-1.5B) and to Countdown and math benchmarks.
• Training is more costly and slower to improve than no-eviction training, and reaches only slightly lower accuracy at full cache scale.
• Evaluation of pass@32 and larger compression depends on a fixed eviction cadence and rate chosen by the designer.

Experimental setup

Datasets: Countdown DAPO-17k

Benchmarks: Countdown AMC 2023 AMC 2025 AIME 2025

Metrics: pass@1 accuracy pass@32 average peak KV cache reduction

Baselines: SnapKV KeyDiff KNorm StreamingLLM Targeted KV dropout Token log-probs only

Builds on: GRPO Dr. GRPO SnapKV Gumbel-top-k

Closest papers by meaning

• Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory (NeurIPS 2026 · similarity 0.88)
• Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models (NeurIPS 2026 · similarity 0.88)
• ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution (ICML 2026 · similarity 0.88)
• Training Large Reasoning Models Efficiently via Progressive Thought Encoding (ICLR 2026 · similarity 0.87)
• Log-Augmented Generation: Scaling Test-Time Reasoning with Reusable Computation (ICLR 2026 · similarity 0.87)
• MemGen: Weaving Generative Latent Memory for Self-Evolving Agents (ICLR 2026 · similarity 0.87)