Research map / Architectures and efficiency / Attention and transformers
Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a hierarchical self-attention (HSA) mechanism for nested multi-scale, multi-modal signals from entropy minimization, computes it in O(M·b²) via dynamic programming, and applies it to transformers trained from scratch and to pretrained RoBERTa zero-shot.
Problem
Standard transformer attention does not natively handle data spanning multiple scales and modalities; existing hierarchical or multi-modal architectures are ad hoc heuristics that either discard geometric information or are hard to generalize.
Approach
Represent data as nested signals with a signal hierarchy, derive attention from entropy minimization of an energy function with a block constraint that ties attention weights across sibling subtrees, and show this is the KL-closest hierarchical approximation to flat Softmax attention; compute it with a two-pass dynamic program.
Key results
- On IMDB with T5-small embeddings, HSA reaches 0.8129 accuracy vs 0.7577 for flat self-attention; on Elec 0.8521 vs 0.8212.
- On N24News, HSA achieves 0.7952 accuracy vs 0.7921 for flat attention and 0.7578 for DeepSet, using about 12M parameters.
- Zero-shot HSA replacement of RoBERTa-base layers 7, 9, 11 cuts attention MFLOPs on IMDB from 214.94 to 4.32 with accuracy dropping from 0.9558 to 0.9494.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HSA instead of flat self-attention | Sentiment classification models · IMDB and Elec (Word2Vec and T5-small embeddings) | accuracy: IMDB Word2Vec: ${0.6739}\pm{0.0004}$ -> 0.7469±0.0029; IMDB T5-small: 0.7577$\pm$0.0024 -> 0.8129$\pm$0.0010; Elec Word2 | Helped |
| Replace Word2Vec token embeddings with T5-small embeddings | FSA and HSA sentiment classifiers · IMDB and Elec | accuracy: IMDB FSA: ${0.6739}\pm{0.0004}$ -> 0.7577$\pm$0.0024; IMDB HSA: 0.7469±0.0029 -> 0.8129$\pm$0.0010; Elec FSA: 0.7182$\pm | Helped |
| Use hierarchical attention and nested multimodal representation instead of flat attention | N24News classifier · N24News | accuracy: FSA: $0.7921\pm 0.0036$; DeepSet: 0.7578; HSA: 0.7952 | Helped |
| Add image modality using the DeepSet architecture | N24News classifier · N24News | accuracy: FSA: $0.7921\pm 0.0036$; DeepSet: 0.7578 | Hurt |
| Replace selected RoBERTa attention layers with HSA, zero-shot | RoBERTa · IMDB, AGNEWS, CoLA, SST-2, MRPC, RTE, and QNLI | accuracy: IMDB: 0.9558 -> 0.9494; AGNEWS: 0.9469 -> 0.9422; CoLA: 0.8150 -> 0.7687; SST-2: 0.9403 -> 0.9025; MRPC: 0.9117 -> 0.855 | Mixed |
| Replace selected RoBERTa attention layers with HSA, zero-shot | RoBERTa · IMDB, AGNEWS, CoLA, SST-2, MRPC, RTE, and QNLI | FLOPs (MFLOPs per impacted layers): IMDB: 214.94 -> 4.32; AGNEWS: 8.99 -> 0.8357; CoLA: 0.4441 -> 0.1912; SST-2: 2.08 -> 0.4132; MRPC: 9.33 -> 0.8481; RTE: | Helped |
| Replace Softmax attention with HSA in all hidden layers | RoBERTa · Benchmark classification tasks | accuracy | Hurt |
| Interleave HSA layers with regular Softmax layers | RoBERTa · Benchmark classification tasks | accuracy | Helped |
| Use HSA in later rather than earlier RoBERTa layers | RoBERTa · Benchmark classification tasks | accuracy | Helped |
Surprising findings
- Zero-shot HSA substitution causes large accuracy drops on some tasks, e.g. QNLI accuracy falls from 0.9267 to 0.5072 for RoBERTa-base.
- DeepSet multi-modal model performs worse than the uni-modal flat-attention baseline on N24News, suggesting how modalities are fused matters.
Limitations
- Zero-shot HSA replacement of all layers degrades performance substantially; fine-tuning is needed to recover accuracy.
- Multi-modal experiments are at moderate scale (~12M parameters) and exclude other tricks used by SOTA methods.
- Large-scale foundation model training and decoder/autoregressive extension are left to future work.
Experimental setup
Datasets: IMDB Elec N24News
Benchmarks: IMDB Elec N24News GLUE (CoLA, SST-2, MRPC, RTE, QNLI) AGNEWS
Metrics: accuracy F1 score precision recall MFLOPs
Baselines: Flat self-attention (Softmax) DeepSet Original RoBERTa
Builds on: Transformer LieTransformer random Fourier features Geometric Deep Learning
Closest papers by meaning
- Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers (NeurIPS 2024 · similarity 0.89)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.88)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.88)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.88)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.