Research map / Architectures and efficiency / Attention and transformers
Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Contract-and-Broadcast Self-Attention (CBSA), derived by unrolling a gradient step of a coding-rate-reduction objective over a few contracted representatives, giving linear-complexity interpretable attention that unifies softmax, linear, and channel attention.
Problem
Attention mechanisms lack clear optimization objectives and suffer quadratic complexity; prior interpretable white-box attentions (MSSA, TSSA) either remain quadratic or degenerate into channel attention.
Approach
Compress all input tokens by contracting a small set of representatives (extracted via cross-attention from pooled initialization within learned orthogonal subspaces), then broadcast the contractions back to tokens; varying the representatives recovers softmax, linear, and channel attention as special cases.
Key results
- CBT-Small reaches 71.4% top-1 on ImageNet-1K with 6.7M params and 4.0G FLOPs, comparable to ViT-S (72.4%) at about 30% of parameters and 40% of FLOPs.
- The best CBT decoder improves over Segmenter by 1.5% mIoU on ADE20K using about 20% of decoder FLOPs and 0.06% of pairwise similarities.
- CBT-S achieves 405 inference images/sec vs 211 for ViT-S on 512x512 CIFAR-10 images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CBSA instead of TSSA | CBT attention on ImageNet-1K · ImageNet-1K | top-1 accuracy: 61.2 -> 63.2; 68.5 -> 71.4 | Helped |
| use CBSA instead of MSSA | CBT attention on ImageNet-1K · ImageNet-1K | top-1 accuracy: 64.7 -> 63.2; 72.1 -> 71.4 | Hurt |
| remove the CBSA contraction step (ablation) | CBSA attention · ImageNet-1K | top-1 accuracy: 63.2 -> 63.8; 71.4 -> 71.8 | Helped |
| combine MSSA in the first half of attention layers with CBSA in the latter half | CBT-S · PASCAL VOC12 validation foreground | IoU: 0.51 -> 0.68 | Helped |
| use learnable representative initialization instead of pooling-based initialization (ablation) | CBSA · CBT models | attention-matrix rank: typically remains around 5; maximal rank can be m = p = 64 | Hurt |
Surprising findings
- Removing the contraction step (Agent Attention variant) in CBSA surpasses linear attention in the adapted-ViT experiments.
- CBT alone loses emergent segmentation properties in later layers, while a hybrid CRATE+CBT model (MSSA in early layers) strengthens them.
Limitations
- Excludes natural language processing tasks because the union-of-linear-subspaces structure does not adapt well to all modalities.
- Performance trails softmax/MSSA baselines in fair comparisons (e.g., CBSA-S 71.4 vs MSSA-S 72.1) and the rigorous CBSA is harder to adapt from pretrained ViTs.
- Over-parameterized projection back to ambient space breaks the theoretical derivation; learnable step sizes are unconstrained in sign.
Experimental setup
Datasets: ImageNet-1K CIFAR10 CIFAR100 Oxford Flowers-102 Oxford-IIIT-Pets ADE20K ImageNet-21k PASCAL VOC12
Benchmarks: ImageNet-1K CIFAR10 CIFAR100 Oxford Flowers-102 Oxford-IIIT-Pets ADE20K PASCAL VOC12
Metrics: top-1 accuracy mIoU FLOPs parameters throughput Jaccard similarity (IoU)
Baselines: ViT CRATE ToST Agent Attention Segmenter DEPICT TSSA MSSA
Builds on: MCR2 CRATE TSSA ISTA Agent Attention
Closest papers by meaning
- Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level (NeurIPS 2024 · similarity 0.87)
- Mixture-of-Top-$k$ Attention: Efficient Attention as Scalable Fast Weights (NeurIPS 2026 · similarity 0.87)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.87)
- Fast attention mechanisms: a tale of parallelism (NeurIPS 2025 · similarity 0.87)
- Attention Mechanism, Max-Affine Partition, and Universal Approximation (NeurIPS 2025 · similarity 0.87)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.