Research map / Architectures and efficiency / Quantization

CompilerKV: Risk-Adaptive KV Cache Compression via Offline Experience Compilation

NeurIPS 2026 · Architectures and efficiency › Quantization · approach: KV cache quantization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CompilerKV, which compiles head-reliability and prompt-risk retention tables offline with conservative Q-learning so that prefill-only KV compression needs only O(1) lookups online beyond the observation-window scan.

Problem

Prefill-only KV cache compression makes irreversible retention decisions, but existing methods estimate per-head reliability and prompt-level compression sensitivity online from a single noisy prompt.

Approach

Treat head reliability and prompt-risk-to-threshold mappings as stable model-level priors compiled offline from a calibration corpus with a horizon-1 conservative Q-learning estimator; at inference, combine a window-aggregated stabilized utility, max-pooled head-weighted scores, and a 20x4 entropy-by-perplexity threshold gate with a budget clamp.

Key results

• At 512-token budget on LongBench, CompilerKV improves over the strongest prefill-only baseline by +1.67 points on average (task-bootstrap 95% CI [+1.08, +2.37]).
• Cross-corpus head-reliability rankings have mean Spearman rho 0.90 across four backbones.
• Direct model-to-model table transfer costs on average 0.54 LongBench points (reported range 0.3 to 0.8).

Surprising findings

• CompilerKV's 512-token compressed LongBench average (42.31 across backbones) slightly exceeds the FullKV reference (42.14), which the authors attribute to denoising rather than information gain.
• Removing the elastic candidate pool / risk gate was the largest single ablation drop, larger than removing stabilized utility or head reliability.

Limitations

• Transfer evaluated only across closely related or similar 7B-8B backbones; more distant architectures and larger-scale transfer are future work.
• Calibration requires a ~50K-prompt offline corpus and per-model compilation of tables (32-layer unified index with depth mapping for other depths).

Experimental setup

Datasets: PG19 arXiv PubMed GovReport QMSum BookSum ShareGPT UltraChat The Pile subsets

Benchmarks: LongBench RULER Needle-in-a-Haystack MATH-500 SQuAD-long SCBench RepoBench-P LongGen GSM8K GPQA

Metrics: average score Spearman rho tokens per second TTFT TPOT peak GPU memory bootstrap 95% CI

Baselines: FullKV SnapKV PyramidKV DynamicKV AdaKV HeadKV CAKE ChunkKV StreamingLLM H2O KVZip FastKV

Builds on: SnapKV CQL DuoAttention

Closest papers by meaning

• EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction (NeurIPS 2026 · similarity 0.90)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.89)
• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.89)
• DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction (NeurIPS 2026 · similarity 0.89)
• Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation (NeurIPS 2026 · similarity 0.89)
• KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation (NeurIPS 2026 · similarity 0.89)