Research map / Architectures and efficiency / Attention and transformers
Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
Jiaqi Leng, Xiang Hu, Junxiong Wang, Jianguo Li, Wei Wu, Yucheng Lu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add one chunk-encoder layer (Enc 0 -> 1) (ablation) | SWA+HSA · RULER avg, ctx 128K | accuracy (%): 40.14 -> 75.47 | Improved |
| add CLS token to 2-layer encoder with bypass (ablation) | SWA+HSA · RULER avg, ctx 32M | accuracy (%): 63.29 -> 79.77 | Improved |
| add CLS token to 1-layer encoder without bypass (ablation) | SWA+HSA · RULER avg, ctx 32M | accuracy (%): 53.94 -> 35.76 | Hurt |
| add Bypassing Residual Path to 2-layer encoder with CLS (ablation) | SWA+HSA · RULER avg, ctx 32M | accuracy (%): 56.18 -> 79.77 | Improved |
| add Bypassing Residual Path to model without encoder (ablation) | SWA+HSA · RULER avg, ctx 128K | accuracy (%): 40.14 -> 72.99 | Improved |
| add encoder layers and CLS with bypass, at training length (ablation) | SWA+HSA · RULER avg, ctx 4K | accuracy (%): 78.12 -> 87.92 | Improved |
| use SWA+HSA instead of Llama-Yarn at 32K | 4K-trained models · RULER avg, ctx 32K | accuracy (%): 0.00 -> 84.86 | Improved |
| increase training context length from 1K to 4K (ablation) | SWA+HSA · RULER avg, ctx 32K | accuracy (%): 17.41 -> 84.86 | Improved |
| increase training length from 4K to 8K (ablation) | SWA+HSA · RULER avg, ctx 32M | accuracy (%): 79.77 -> 61.67 at 32M; 85.65 -> 93.94 at 128K | Mixed |
| use Top-K 8 instead of Top-K 64 (ablation) | SWA+HSA · RULER avg, ctx 32M | accuracy (%): 69.70 -> 79.77 | Improved |
| retrieve every 8 tokens instead of every token (ablation) | SWA+HSA · RULER avg, ctx 128K | accuracy (%): 85.65 -> 54.55 | Hurt |
| replace Transformer attention with SWA+HSA in 8B MoE | Transformer-MoE · general benchmarks (avg) | average score: 55.62 -> 57.69 | Improved |
Closest papers by meaning
- Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling (ICML 2025 · similarity 0.91)
- Training-Free Long-Context Scaling of Large Language Models (ICML 2024 · similarity 0.90)
- The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.90)
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.90)
- Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models (NeurIPS 2026 · similarity 0.89)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
