Research map / Architectures and efficiency / Attention and transformers
Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SkipV1Former, which reuses half of each layer's Value heads from the first layer to cut Value projections and V cache by ~50% (~25% total KV cache) while improving perplexity, with a theoretical mesa-optimization analysis and an uptraining recipe.
Problem
Transformers' KV cache and parameter costs grow with scale, while existing skip connections either improve representation without reducing KV cache or reduce KV cache at the cost of representation quality.
Approach
From layer 2 onward, each layer replaces half of its Value heads with the corresponding first-layer Value heads (a cross-layer Value skip), so the V cache and V projections shrink by about half while the first layer's uncompressed Values restore information lost to copying in deeper layers.
Key results
- SkipV1Former reduces KV cache by ~25% across scales while improving validation loss over MHA by roughly 0.03-0.05 on 60M-350M LLaMA models.
- On 3B LLaMA, SkipV1Former averages 48.3% vs 47.5% for baseline on nine downstream tasks.
- Uptraining from GPT-2 MHA checkpoints matches from-scratch loss using 10-15% of original compute; SkipV1Former-MLA reduces KV bytes per token by 44.4% vs MLA with lower val loss (2.888 vs 2.896).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add first-layer Value skip connections to GQA | Transformer-GQA · GPT2-355M on OpenWebText2 | validation loss: 2.912 -> 2.893 | Helped |
| Add first-layer Value skip connections to MLA | Transformer-MLA · GPT2-355M on OpenWebText2 | validation loss: 2.896 -> 2.888 | Helped |
| Add first-layer Value skip connections to GQA | Transformer-GQA · GPT2-355M on OpenWebText2 | KV bytes per token: 98 304 B -> 74 752 B | Helped |
| Add first-layer Value skip connections to MLA | Transformer-MLA · GPT2-355M on OpenWebText2 | KV bytes per token: 13 824 B -> 7 680 B | Helped |
| Use fewer than 50% skip heads (ablation) | SkipV1Former · GPT2-355M | validation loss | Hurt |
| Use more than 50% skip heads (ablation) | SkipV1Former · GPT2-355M | validation loss | Hurt |
| Remove Value skip connections while halving the Value projection (ablation) | SkipV1Former · GPT2-355M | validation loss | Hurt |
| Reuse first-layer Values rather than second-layer Values (ablation) | SkipV1Former · GPT2-355M | validation loss | Helped |
| Reuse first-layer Values rather than third- or fourth-layer Values (ablation) | SkipV1Former · GPT2-355M | validation loss | Helped |
Surprising findings
- Reusing second-layer Values (SkipV2) gives only marginal gains, and halving Value projections without any skip (SkipV0) severely degrades performance.
- SkipV1Former fails to boost performance on non-autoregressive tasks, per the appendix.
Limitations
- Theoretical analysis uses a simplified two-layer linear attention model without residual connections and a copying-mechanism assumption.
- Experiments are limited to small-to-medium scales (up to 3B pretraining, 7B only for inference measurement), and downstream results are near chance for many tasks.
- Current implementation lacks fused kernels, so the prefill and throughput overheads are not fully optimized.
Experimental setup
Datasets: OpenWebText2 C4
Benchmarks: OpenWebText2 validation loss C4 validation loss zero-shot downstream tasks (ARC-C, ARC-E, BoolQ, HellaSwag, OBQA, PIQA, RTE, SciQ, Winogrande)
Metrics: validation loss perplexity accuracy KV cache size GPU memory prefill time throughput
Baselines: MHA Transformer DenseFormer ResFormer YOCO Cross-Layer Attention GQA MLA
Builds on: ResFormer YOCO Multi-Query Attention DenseFormer
Closest papers by meaning
- Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (NeurIPS 2024 · similarity 0.90)
- MemoryFormer : Minimize Transformer Computation by Removing Fully-Connected Layers (NeurIPS 2024 · similarity 0.88)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.88)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.87)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.87)
- SALS: Sparse Attention in Latent Space for KV Cache Compression (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.