Research map / Architectures and efficiency / Attention and transformers
Head-in-Head in Linear Attention
Shijie Mei, Man Yao, Jiabo Tong, Bo XU, Guoqi Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Head-in-Head intra-head mask grouping (r=4, data-dependent) to Gated-DeltaNet | Gated-DeltaNet 1.3B · Wikitext / LAMBADA | perplexity: 16.30 -> 16.10 (Wiki); 14.52 -> 13.05 (Lamb.) | Improved |
| add Head-in-Head to Gated-DeltaNet | Gated-DeltaNet 340M · Commonsense reasoning avg | accuracy (%): 41.79 -> 42.43 | Improved |
| add Head-in-Head to Gated-DeltaNet | Gated-DeltaNet 340M · Recall-intensive tasks | average score: 24.88 -> 27.28 | Improved |
| add Head-in-Head to DeltaNet / Gated-DeltaNet | DeltaNet, Gated-DeltaNet · MAD benchmark | average score: 74.4 -> 76.5 (DN); 74.9 -> 77.2 (GDN) | Improved |
| add Head-in-Head to Gated-DeltaNet | Gated-DeltaNet 1.3B · LongBench | average score: 17.06 -> 17.57 | Improved |
| increase intra-head groups from r=1 to r=2 and r=4 (static mask) (ablation) | Gated-DeltaNet 1.3B · CSR-avg | accuracy (%): 51.76 -> 52.17 -> 52.24 | Improved |
| increase intra-head groups from r=2 to r=4 (ablation) | Gated-DeltaNet 1.3B · Recall-avg | recall average: 36.48 -> 37.62 | Improved |
| switch from static learnable mask to data-dependent (time-varying) mask at r=4 (ablation) | Head-in-Head GDN 1.3B · Wiki/Lamb perplexity vs Recall-avg | perplexity and recall: Lamb 13.61 -> 13.05; Recall-avg 37.62 -> 37.07 | Mixed |
| add Head-in-Head grouping (more intra-head groups) to GDN (ablation) | Gated-DeltaNet · PG-19 memory-state similarity | information entropy: 0.756 -> 0.813 (rank2) -> 0.823 (rank4) | Improved |
| add Head-in-Head to Gated-DeltaNet: effect on length extrapolation | Gated-DeltaNet 1.3B · PG-19 up to 16k | perplexity | Improved |
| add time-varying mask vs fixed mask: inference latency (ablation) | HH-GDN 1.3B · decoding 1000 tokens | latency relative to baseline: 1.2x (time-varying), 1.05x (fixed) | Hurt |
Closest papers by meaning
- In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention (ICML 2025 · similarity 0.87)
- Virtual Head Attention (NeurIPS 2026 · similarity 0.87)
- LinearARD: Linear-Memory Attention Distillation for RoPE Restoration (NeurIPS 2026 · similarity 0.86)
- Bayes optimal learning of attention-indexed models (NeurIPS 2025 · similarity 0.86)
- Memorization Capacity of Multi-Head Attention in Transformers (ICLR 2024 · similarity 0.86)
- MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
