Research map / Architectures and efficiency / Attention and transformers
FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension
Jushi Kai, Yixuan Wang, Boyi Zeng, Haoli Bai, Bo Jiang, Ziwei He, Zhouhan Lin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| keep low-frequency instead of high-frequency components in KV compression (ablation) | FreqKV · LongBench GovReport | score: 14.21 -> 25.51 | Improved |
| keep low-frequency instead of high-frequency components in KV compression (ablation) | FreqKV · PG-19 validation PPL, 16384 | perplexity: 8.70 -> 7.73 | Improved |
| compress key states before RoPE rather than after (ablation) | FreqKV · PG-19 test PPL, eval 8192 | perplexity: 7.13 -> 7.04 | Improved |
| add the sqrt(L/N) rescaling factor after IDCT (ablation) | FreqKV · training loss | training loss | Improved |
| compress in frequency domain instead of dropping old tokens (Dropping) (ablation) | FreqKV · Needle-in-a-Haystack, LLaMA-2-chat-7B | retrieval accuracy | Improved |
| increase sink tokens from 0 to 4 (ablation) | FreqKV · PG-19 validation PPL, 8192 | perplexity: 7.84 -> 7.76 | Improved |
| increase sink tokens from 4 to 8 (ablation) | FreqKV · PG-19 validation PPL, 8192 | perplexity: 7.76 -> 7.75 | No clear effect |
| retain more frequency components (larger retaining ratio) (ablation) | FreqKV · PG-19 validation PPL | perplexity and latency | Mixed |
| use larger retaining ratio at evaluation than in training (ablation) | FreqKV · PG-19 validation PPL, 8K | perplexity: 7.76 -> 7.77 (train 0.5, eval 0.5 vs 0.75) | Improved |
| apply FreqKV compression instead of eviction for inputs beyond the native window | LLaMA-3-instruct-8B · RULER 16K average | average score: 0.10 -> 47.30 | Improved |
| use FreqKV instead of full FT/LongLoRA at long evaluation lengths on Proof-pile | FreqKV · Proof-pile PPL, 7B trained 32K, eval 32768 | perplexity: 2.50 -> 2.72 | Hurt |
| use FreqKV instead of full-attention fine-tuning for training | LLaMA-2-7B · training cost at 8K | training time (hours): 28.88 -> 17.07 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- LoCoCo — LoCoCo: Dropping In Convolutions for Long Context Compression (ICML 2024) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.92)
- Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs (NeurIPS 2025 · similarity 0.91)
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.91)
- PatchKV: Weight Space Compensation of KV Cache (NeurIPS 2026 · similarity 0.91)
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.91)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
