Research map / Architectures and efficiency / Attention and transformers
Reconstructing KV Caches with Cross-Layer Fusion for Enhanced Transformers
Hongzhan Lin, ZhiqiBai, Xinmiao Zhang, Siran Yang, Jiamang Wang, Yunlong Xu, Jiaheng Liu, Yongchi Zhao, Xiang Li, Yuchi Xu, Wenbo Su, Bo Zheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fuse bottom and middle layer KV caches into top layers instead of full cache | Vanilla Transformer · FineWeb-Edu valid loss, 1.5B | valid loss: 2.241 -> 2.221 | Improved |
| fuse bottom and middle layer KV caches into top layers instead of full cache | Vanilla Transformer · Downstream average, 1.5B | average accuracy: 54.55 -> 55.82 | Improved |
| fuse KV caches (FusedKV) instead of full cache | Vanilla Transformer · FineWeb-Edu valid loss, 4B | valid loss: 2.002 -> 1.978 | Improved |
| fuse KV caches (FusedKV) instead of full cache | Vanilla Transformer · Downstream average, 4B | average accuracy: 59.07 -> 60.01 | Improved |
| use FusedKV instead of CLA at equal 50% cache | CLA · Downstream average, 650M | average accuracy: 46.49 -> 47.45 | Improved |
| use FusedKV instead of YOCO at equal 50% cache | YOCO · Downstream average, 332M | average accuracy: 43.08 -> 43.59 | Improved |
| use FusedKV instead of GQA at equal 50% cache | GQA · Downstream average, 1.5B | average accuracy: 54.58 -> 55.82 | Improved |
| use FusedKV-Lite sharing instead of FusedKV (ablation) | FusedKV · Downstream average, 650M | average accuracy: 47.45 -> 48.51 | Improved |
| reverse KV assignment (K from bottom, V from middle) (ablation) | FusedKV-Lite · 650M, 200B tokens, validation loss and Hellaswag | valid loss, accuracy | Hurt |
| add learnable channel-wise weights to FusedKV-Lite (ablation) | FusedKV-Lite · 650M, WikiText and LAMBADA perplexity | perplexity | Improved |
| apply FusedKV on top of MLA | MLA · WikiText, 650M | perplexity: 21.03 -> 20.08 | Improved |
| apply FusedKV on top of MLA | MLA · downstream average, 650M | average accuracy: 45.03 -> 44.79 | No clear effect |
Closest papers by meaning
- Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads (NeurIPS 2025 · similarity 0.86)
- Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (NeurIPS 2024 · similarity 0.85)
- Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse (NeurIPS 2026 · similarity 0.85)
- InfoFlow: A Framework for Multi-Layer Transformer Analysis (NeurIPS 2026 · similarity 0.85)
- Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching (NeurIPS 2024 · similarity 0.84)
- DeepCrossAttention: Supercharging Transformer Residual Connections (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
