AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction

Xiaohan Qin, Xiangdong Zhang, Yu Wang, Huaijin Wu, Zhuo Xia, Yebin Yang, Junchi Yan

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DepthGraft, a cross-layer KV sharing scheme with hierarchical KV reconstruction to counter depth-wise information dilution in Pre-LN LLMs without extra computation or memory access.

Problem

Pre-LN LLMs suffer from depth-wise information dilution, limiting the effective use of deep layers; existing cross-layer connectivity designs add computation or memory access.

Approach

Revisits cross-layer KV sharing as an efficient form of cross-layer connectivity and extends it with hierarchical cross-layer KV reconstruction to regularize deep-layer structure. Only the abstract was available, so further specifics are not stated.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
enable gradient flow through shared KV (vs detach) (ablation)Shallow-Copy · 16B A3B MoE ablation (7 benchmarks avg)average score: 42.86 -> 45.30Improved
detach shared KV (no cross-layer gradients) (ablation)Shallow-Copy · 16B A3B MoE ablation (7 benchmarks avg)average score: 42.38 -> 42.86No clear effect
reconstruct from shallow+middle source pair vs single source copy (ablation)DepthGraft · 16B A3B MoE ablation (7 benchmarks avg)average score: 45.30 -> 47.56Improved
use 3-stage layout instead of 2-stage (ablation)DepthGraft · 16B A3B MoE ablation (7 benchmarks avg)average score: 43.58 -> 47.56Improved
use 3-stage two-source instead of 4-stage three-source (ablation)DepthGraft · 16B A3B MoE ablation (7 benchmarks avg)average score: 45.40 -> 47.56Improved
use 3-stage instead of 4-stage two-source (ablation)DepthGraft · 16B A3B MoE ablation (7 benchmarks avg)average score: 45.33 -> 47.56Improved
adjacent-layer KV sharing (CLA/MLKV)GQA 16B MoE · 16B A3B MoE ablation (7 benchmarks avg)average score: 42.38 -> 38.05Hurt
global-anchor sharing (YOCO)GQA 16B MoE · 16B A3B MoE ablation (7 benchmarks avg)average score: 42.38 -> 43.33No clear effect
DepthGraft vs FusedKVKV sharing 16B MoE · 16B A3B MoE ablation (7 benchmarks avg)average score: 45.64 -> 47.56Improved
add DepthGraft to dense modelsGQA dense 0.5B-1.5B · Avg over 12 benchmarksaverage score: 28.36 -> 30.78 (1.5B)Improved
add DepthGraft to MoEGQA/MLA MoE 3B-16B · Avg over 12 benchmarksaverage score: 43.43 -> 47.04 (16B)Improved
add DepthGraft16B MoE decoding · TPOTms: 7.79 -> 7.84 (128K)No clear effect

Limitations

Experimental setup

Builds on: Pre-LN transformer

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy