Research map / Architectures and efficiency / Attention and transformers

$\text{D}_{2}\text{O}$: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models

ICLR 2025 · Architectures and efficiency › Attention and transformers · approach: KV cache eviction and compression

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes D2O, a training-free KV cache compression method combining layer-wise dynamic budget allocation (inverse-variance softmax of attention density) with token eviction plus EMA-thresholded similarity-based merging of evicted tokens.

Problem

KV cache memory grows linearly with context length in LLM generative inference, and existing eviction methods discard tokens irreversibly and use uniform per-layer budgets, causing context loss and hallucinations on long contexts.

Approach

Allocates per-layer KV cache budgets via an inverse-variance softmax over prompt attention density, and instead of discarding evicted tokens, merges those whose key similarity to retained tokens exceeds an EMA-smoothed threshold into their nearest retained token using similarity-weighted fusion; no fine-tuning required.

Key results

• On LongBench at 20% KV budget, D2O beats H2O on Llama-3-8B across tasks (e.g., average gains of several points; MF-en 19.93 vs 17.22 for H2O, GovReport 24.36 vs 21.29).
• Throughput reaches up to 3.04x the full model at 2048+8192 with Llama-3-8B on A100-80G, with batch size 4 vs 1 for the full model.
• On Needle-in-a-Haystack at 100k with 8192 budget, D2O scores 91.88 vs 72.34 for H2O and 78.88 for CaM (full model 94.46).

Measured effects reported in the paper

Surprising findings

• On TruthfulQA, D2O even surpasses the full KV cache model across most budget ratios on the four Llama backbones.
• Merging keys for similarity works much better than using values (CoQA 59.25 vs 53.88 for values), and separate key/value similarity hurts performance.

Limitations

• Merging step adds some computational overhead compared with pure eviction (H2O), per the authors.
• Hyperparameters (beta, N:M ratio) are tuned on a few datasets (TREC, CoQA, GSM8K, TruthfulQA) and may not generalize across all tasks.
• Full-cache baseline runs out of memory on ultra-long multi-turn dialogues, so some comparisons are limited to eviction baselines.

Experimental setup

Datasets: GSM8K CoQA TruthfulQA LongBench Needle In A Haystack PG-19 MT-bench InfiniteBench longbook_sum_eng

Benchmarks: GSM8K CoQA TruthfulQA LongBench Needle In A Haystack PG-19 InfiniteBench

Metrics: exact match accuracy BLEU F1 / task score perplexity (NLL) throughput (tokens/s) Rouge-Lsum

Baselines: StreamingLLM H2O RoCo PyramidKV CaM SnapKV Local Window

Builds on: H2O StreamingLLM CaM

Closest papers by meaning

• LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models (ICML 2025 · similarity 0.91)
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.91)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.91)
• Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.90)
• ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction (NeurIPS 2024 · similarity 0.90)
• MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.90)