AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

DSSA: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation

Weilin Zhao, Zihan Zhou, Zhou su, Chaojun Xiao, Yuxuan Li, Yanghao Li, Yudi Zhang, Weilun Zhao, Zhen Li, Yuxiang Huang, Ao Sun, Xu Han and 1 more

ICLR 2026 · Architectures and efficiency › Attention and transformers · approach: Linear attention approximations

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use InfLLM-V2 shared-parameter sparse attention instead of NSA for short-to-long adaptation8B model · RULER 32kaverage accuracy: 59.92 -> 82.62Improved
use InfLLM-V2 instead of NSA8B model · LongPPLperplexity: 4.24 -> 2.12Improved
use InfLLM-V2 instead of NSA8B model · long reasoning averageaverage: 37.28 -> 42.66Improved
switch InfLLM-V2 to dense mode at inference (ablation)InfLLM-V2 · RULER 32kaverage accuracy: 82.62 -> 88.32Improved
add LSE Approximation to block selection (ablation)InfLLM-V2 (Sparse) · RULER 32kaverage accuracy: 82.09 -> 82.62No clear effect
add LSE Approximation to block selection (ablation)InfLLM-V2 kernel · Block selection time, A100 128ktime (ms): 75.36 -> 56.59Improved
return to dense attention after long finetuning with InfLLM-V2InfLLM-V2 (Dense) · general short tasks averageaverage: 67.41 -> 66.76No clear effect
use NSA instead of full attention during long finetuning8B model · general short tasks averageaverage: 67.41 -> 60.63Hurt
InfLLM-V2 Sparse vs FullAttn on long reasoning8B model · AIME 25accuracy: 30.63 -> 29.38No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy