AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition

Zhengfu He, Junxuan Wang, Rui Lin, Xuyang Ge, Wentao Shu, Qiong Tang, Junping Zhang, Xipeng Qiu

ICLR 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 6 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fix Lorsa QK parameters to the original MHSA QK instead of learning them (ablation)Lorsa · QK initialization, Pythia-160MFVU (%): 11.3% -> 12.4%Hurt
initialize Lorsa QK from MHSA QK (trainable) instead of random (ablation)Lorsa · QK initialization, Pythia-160MFVU (%): 11.3% -> 11.2%No clear effect
reduce Lorsa QK dimension below MHSA head dimension 64 (ablation)Lorsa · Pythia-160M, context 256 and 1024reconstruction performanceHurt
bind QK across Lorsa heads while keeping independent QK count at least the MHSA head count (ablation)Lorsa · Pythia-160Mreconstruction performance and QK costImproved
bind QK so aggressively that independent QK heads fall below the MHSA head count (ablation)Lorsa · Pythia-160Mreconstruction performanceHurt
replace SAE with Lorsa for interpretability of attentionLorsa vs attention SAE · Pythia-160M autointerpautointerp score: 6 wins, 3 losses, 15 ties of 24 comparisonsNo clear effect
add ReLU to Lorsa activations z (ablation)Lorsa · training dynamicstraining lossNo clear effect
avoid grouped queries in Lorsa for GQA models (ablation)Lorsa · Pythia-160M and Llama-3.1-8BperformanceNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy