Research map / Architectures and efficiency / Attention and transformers
Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates Mamba, Mamba-2, Griffin/Hawk, RWKV (and RetNet, HGRN in appendix) as single data-controlled implicit causal attention matrices, and uses them for attribution-based explainability and weakly supervised segmentation.
Problem
Attention-free gated linear RNNs (Mamba, RWKV, Griffin, etc.) lack a unified attention view, and the prior implicit-attention formulation for Mamba ignored gating, Conv1D and other sub-layers, limiting their use for explainability.
Approach
Write each gated-linear RNN layer as y = H x with a data-dependent matrix H that folds in the S6/RG-LRU/WKV core, short convolutions, gate branches, activations and normalization as diagonal and convolution matrices, then apply off-the-shelf Transformer explainability methods to H.
Key results
- On ImageNet positive perturbation, the proposed Mamba attribution reaches AUC 11.350 vs 16.619 for the prior Mamba attribution (lower is better).
- On ImageNet-Segmentation, Mamba-Attr (ours) reaches 62.51 mIoU vs 54.24 for prior Mamba-Attr with ViM-S.
- On pruning perturbation for Mamba 2.8B, AU-MSE drops from 1.765 (prior) to 1.082 (ours); AMPLIFY with the proposed attribution averages 1.2 points above the prior AMPLIFY baseline on ICL tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed implicit-attention attribution instead of the previous Mamba attention formulation | Mamba attribution · ImageNet validation, positive perturbation | AUC: 16.619 -> 11.350 | Helped |
| Use the proposed implicit-attention attribution instead of the previous Mamba attention formulation | Mamba attribution · ImageNet validation, negative perturbation | AUC: 39.632 -> 51.310 | Helped |
| Use the proposed Mamba attribution method instead of the previous Mamba attribution | ViM-S segmentation explanations · ImageNet-Segmentation | mIoU: 54.24 -> 62.51 | Helped |
| Use the proposed attribution method with AMPLIFY instead of the previous attribution method | Mamba 1.3B · Snarks | accuracy: 53.11 -> 56.15 | Helped |
| Use the proposed attribution method with AMPLIFY instead of the previous attribution method | Mamba 1.3B · CommonsenseQA | accuracy: 53.55 -> 54.72 | Helped |
| Use the proposed attribution method with AMPLIFY instead of the previous attribution method | Mamba 1.3B · Formal Fallacies | accuracy: 44.28 -> 45.22 | Helped |
| Use the proposed attribution method with AMPLIFY instead of the previous attribution method | Mamba 2.8B · Snarks | accuracy: 58.10 -> 60.02 | Helped |
| Use the proposed attribution method with AMPLIFY instead of the previous attribution method | Mamba 2.8B · CommonsenseQA | accuracy: 56.10 -> 56.08 | Hurt |
| Use the proposed attribution method with AMPLIFY instead of the previous attribution method | Mamba 2.8B · Formal Fallacies | accuracy: 47.80 -> 47.86 | Helped |
| Use Mamba-Attr instead of DeiT-S CAMs to generate pseudo-labels | Weakly supervised semantic segmentation · Pascal-VOC 2012 | mIoU: 35.99 -> 52.11 | Helped |
| Include Conv1D in the implicit-attention representation (ablation) | Mamba attribution · Attribution perturbation tests | performance | Helped |
| Include the gate branch in the implicit-attention representation (ablation) | Mamba attribution · Attribution perturbation tests | performance | Helped |
Limitations
- Explanation quality is evaluated only through proxy tests (perturbation, segmentation) rather than ground-truth faithfulness.
- Formulations for RetNet and HGRN are given only in the appendix and are partly incomplete or repetitive.
- Only a subset of architectures (Mamba, Mamba-2, Griffin/Hawk, RWKV) receive full treatment.
Experimental setup
Datasets: ImageNet ImageNet-Segmentation ARC-E IMDB Pascal VOC 2012 Snarks CommonsenseQA Formal Fallacies
Benchmarks: ImageNet ImageNet-Segmentation ARC-E IMDB Pascal VOC 2012 Snarks CommonsenseQA Formal Fallacies
Metrics: AUC (positive/negative perturbation) AUAC AU-MSE pixel accuracy mAP mIoU accuracy
Baselines: Mamba implicit attention (Ali et al.) ViT / DeiT explainability (attention rollout, Transformer-Attribution) CAM Toco BERT explainability AMPLIFY
Builds on: Ali et al. implicit attention for Mamba (S6) Transformer explainability (attention rollout, attribution) Hyena AMPLIFY
Closest papers by meaning
- Learning to (Learn at Test Time): RNNs with Expressive Hidden States (ICML 2025 · similarity 0.89)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.87)
- Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks (NeurIPS 2024 · similarity 0.87)
- Gated Linear Attention Transformers with Hardware-Efficient Training (ICML 2024 · similarity 0.86)
- Linear Attention for Efficient Bidirectional Sequence Modeling (NeurIPS 2025 · similarity 0.86)
- Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.