Research map / Architectures and efficiency / RNNs and normalization
DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction
Xiaohan Qin, Xiangdong Zhang, Yu Wang, Huaijin Wu, Zhuo Xia, Yebin Yang, Junchi Yan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DepthGraft, a cross-layer KV sharing scheme with hierarchical KV reconstruction to counter depth-wise information dilution in Pre-LN LLMs without extra computation or memory access.
Problem
Pre-LN LLMs suffer from depth-wise information dilution, limiting the effective use of deep layers; existing cross-layer connectivity designs add computation or memory access.
Approach
Revisits cross-layer KV sharing as an efficient form of cross-layer connectivity and extends it with hierarchical cross-layer KV reconstruction to regularize deep-layer structure. Only the abstract was available, so further specifics are not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| enable gradient flow through shared KV (vs detach) (ablation) | Shallow-Copy · 16B A3B MoE ablation (7 benchmarks avg) | average score: 42.86 -> 45.30 | Improved |
| detach shared KV (no cross-layer gradients) (ablation) | Shallow-Copy · 16B A3B MoE ablation (7 benchmarks avg) | average score: 42.38 -> 42.86 | No clear effect |
| reconstruct from shallow+middle source pair vs single source copy (ablation) | DepthGraft · 16B A3B MoE ablation (7 benchmarks avg) | average score: 45.30 -> 47.56 | Improved |
| use 3-stage layout instead of 2-stage (ablation) | DepthGraft · 16B A3B MoE ablation (7 benchmarks avg) | average score: 43.58 -> 47.56 | Improved |
| use 3-stage two-source instead of 4-stage three-source (ablation) | DepthGraft · 16B A3B MoE ablation (7 benchmarks avg) | average score: 45.40 -> 47.56 | Improved |
| use 3-stage instead of 4-stage two-source (ablation) | DepthGraft · 16B A3B MoE ablation (7 benchmarks avg) | average score: 45.33 -> 47.56 | Improved |
| adjacent-layer KV sharing (CLA/MLKV) | GQA 16B MoE · 16B A3B MoE ablation (7 benchmarks avg) | average score: 42.38 -> 38.05 | Hurt |
| global-anchor sharing (YOCO) | GQA 16B MoE · 16B A3B MoE ablation (7 benchmarks avg) | average score: 42.38 -> 43.33 | No clear effect |
| DepthGraft vs FusedKV | KV sharing 16B MoE · 16B A3B MoE ablation (7 benchmarks avg) | average score: 45.64 -> 47.56 | Improved |
| add DepthGraft to dense models | GQA dense 0.5B-1.5B · Avg over 12 benchmarks | average score: 28.36 -> 30.78 (1.5B) | Improved |
| add DepthGraft to MoE | GQA/MLA MoE 3B-16B · Avg over 12 benchmarks | average score: 43.43 -> 47.04 (16B) | Improved |
| add DepthGraft | 16B MoE decoding · TPOT | ms: 7.79 -> 7.84 (128K) | No clear effect |
Limitations
- Only the abstract was available; experimental details unknown.
Experimental setup
Builds on: Pre-LN transformer
Closest papers by meaning
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.87)
- Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse (NeurIPS 2026 · similarity 0.87)
- Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads (NeurIPS 2025 · similarity 0.86)
- Do Language Models Use Their Depth Efficiently? (NeurIPS 2025 · similarity 0.86)
- EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction (NeurIPS 2026 · similarity 0.86)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
