Research map / Architectures and efficiency / Attention and transformers
Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
Zhengfu He, Junxuan Wang, Rui Lin, Xuyang Ge, Wentao Shu, Qiong Tang, Junping Zhang, Xipeng Qiu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fix Lorsa QK parameters to the original MHSA QK instead of learning them (ablation) | Lorsa · QK initialization, Pythia-160M | FVU (%): 11.3% -> 12.4% | Hurt |
| initialize Lorsa QK from MHSA QK (trainable) instead of random (ablation) | Lorsa · QK initialization, Pythia-160M | FVU (%): 11.3% -> 11.2% | No clear effect |
| reduce Lorsa QK dimension below MHSA head dimension 64 (ablation) | Lorsa · Pythia-160M, context 256 and 1024 | reconstruction performance | Hurt |
| bind QK across Lorsa heads while keeping independent QK count at least the MHSA head count (ablation) | Lorsa · Pythia-160M | reconstruction performance and QK cost | Improved |
| bind QK so aggressively that independent QK heads fall below the MHSA head count (ablation) | Lorsa · Pythia-160M | reconstruction performance | Hurt |
| replace SAE with Lorsa for interpretability of attention | Lorsa vs attention SAE · Pythia-160M autointerp | autointerp score: 6 wins, 3 losses, 15 ties of 24 comparisons | No clear effect |
| add ReLU to Lorsa activations z (ablation) | Lorsa · training dynamics | training loss | No clear effect |
| avoid grouped queries in Lorsa for GQA models (ablation) | Lorsa · Pythia-160M and Llama-3.1-8B | performance | No clear effect |
Closest papers by meaning
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.89)
- Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers (NeurIPS 2025 · similarity 0.89)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.88)
- What are you sinking? A geometric approach on attention sink (NeurIPS 2025 · similarity 0.88)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.88)
- The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
