AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models

Jiaqi Leng, Xiang Hu, Junxiong Wang, Jianguo Li, Wei Wu, Yucheng Lu

ICLR 2026 · Architectures and efficiency › Attention and transformers · approach: Linear attention approximations · 5 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add one chunk-encoder layer (Enc 0 -> 1) (ablation)SWA+HSA · RULER avg, ctx 128Kaccuracy (%): 40.14 -> 75.47Improved
add CLS token to 2-layer encoder with bypass (ablation)SWA+HSA · RULER avg, ctx 32Maccuracy (%): 63.29 -> 79.77Improved
add CLS token to 1-layer encoder without bypass (ablation)SWA+HSA · RULER avg, ctx 32Maccuracy (%): 53.94 -> 35.76Hurt
add Bypassing Residual Path to 2-layer encoder with CLS (ablation)SWA+HSA · RULER avg, ctx 32Maccuracy (%): 56.18 -> 79.77Improved
add Bypassing Residual Path to model without encoder (ablation)SWA+HSA · RULER avg, ctx 128Kaccuracy (%): 40.14 -> 72.99Improved
add encoder layers and CLS with bypass, at training length (ablation)SWA+HSA · RULER avg, ctx 4Kaccuracy (%): 78.12 -> 87.92Improved
use SWA+HSA instead of Llama-Yarn at 32K4K-trained models · RULER avg, ctx 32Kaccuracy (%): 0.00 -> 84.86Improved
increase training context length from 1K to 4K (ablation)SWA+HSA · RULER avg, ctx 32Kaccuracy (%): 17.41 -> 84.86Improved
increase training length from 4K to 8K (ablation)SWA+HSA · RULER avg, ctx 32Maccuracy (%): 79.77 -> 61.67 at 32M; 85.65 -> 93.94 at 128KMixed
use Top-K 8 instead of Top-K 64 (ablation)SWA+HSA · RULER avg, ctx 32Maccuracy (%): 69.70 -> 79.77Improved
retrieve every 8 tokens instead of every token (ablation)SWA+HSA · RULER avg, ctx 128Kaccuracy (%): 85.65 -> 54.55Hurt
replace Transformer attention with SWA+HSA in 8B MoETransformer-MoE · general benchmarks (avg)average score: 55.62 -> 57.69Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy