Research map / Architectures and efficiency / Attention and transformers
Attention (as Discrete-Time Markov) Chains
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Interprets each self-attention matrix as a discrete-time Markov chain and proposes multi-bounce attention, TokenRank (steady-state vector), and lambda_2 head weighting, showing gains in zero-shot segmentation, SAG image generation, DiffSeg, and token masking.
Problem
Common attention operations (row/column select, column sums, head averaging) only capture direct attention effects and ignore indirect paths and noise in attention maps when used for visualization, explainability, and downstream tasks in vision transformers.
Approach
Treat row-normalized attention as a DTMC; iterating the selection (power method) yields multi-bounce attention capturing indirect effects, the steady-state vector (TokenRank) gives global token importance, and head weighting by the second-largest eigenvalue lambda_2 favors heads with metastable structure.
Key results
- Zero-shot ImageNet segmentation: Ours achieves 84.12 Acc, 70.20 mIoU, 94.29 mAP, versus ConceptAttention's 83.07/71.04/90.45, with a large mAP gap.
- SAG+TokenRank improves IS from 17.69 to 18.37 and FID from 52.48 to 50.14 over SAG.
- On COCO-Stuff-27, TokenRank grid anchors raise DiffSeg mACC from 72.50 to 84.97 (mIoU 43.60 to 44.87).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add λ₂ weighting to attention-head averaging (ablation) | Multi-bounce segmentation on FLUX DiT · ImageNet Segmentation | Acc: 84.00 -> 84.12 | Helped |
| Add λ₂ weighting to attention-head averaging (ablation) | Multi-bounce segmentation on FLUX DiT · ImageNet Segmentation | mIoU: 70.02 -> 70.20 | Helped |
| Add λ₂ weighting to attention-head averaging (ablation) | Multi-bounce segmentation on FLUX DiT · ImageNet Segmentation | mAP: 94.28 -> 94.29 | Helped |
| Use dual-channel instead of single-channel attention blocks (ablation) | Multi-bounce segmentation on FLUX DiT · ImageNet Segmentation | Acc: 69.63 -> 84.12 | Helped |
| Use dual-channel instead of single-channel attention blocks (ablation) | Multi-bounce segmentation on FLUX DiT · ImageNet Segmentation | mIoU: 51.73 -> 70.20 | Helped |
| Use dual-channel instead of single-channel attention blocks (ablation) | Multi-bounce segmentation on FLUX DiT · ImageNet Segmentation | mAP: 80.50 -> 94.29 | Helped |
| Use two outgoing bounces instead of three (ablation) | Multi-bounce segmentation on FLUX DiT · ImageNet Segmentation | mAP: 89.80 -> 94.29 | Helped |
| Use three incoming bounces instead of two (ablation) | Multi-bounce segmentation on FLUX DiT · ImageNet Segmentation | mAP: 93.68 -> 94.01 | Helped |
| Use three incoming bounces instead of two (ablation) | Multi-bounce segmentation on FLUX DiT · ImageNet Segmentation | Acc: 83.73 -> 82.93 | Hurt |
| Replace SAG column-sum token selection with TokenRank | SAG with SD1.5 · Unconditional image generation (50K images) | IS: 17.69 -> 18.37 | Helped |
| Replace SAG column-sum token selection with TokenRank | SAG with SD1.5 · Unconditional image generation (50K images) | FID: 52.48 -> 50.14 | Helped |
| Replace uniform-grid anchors with TokenRank-selected anchors | DiffSeg · COCO-Stuff-27 | mACC: 72.50 -> 84.97 | Helped |
Surprising findings
- Outgoing (column-based) multi-bounce attention degrades rapidly with more bounces, as steady state emphasizes background hubs.
- Random noisy attention matrices converge to a dispersed map after one step, while structured attention keeps consolidating metastable regions.
Limitations
- Restricted to square attention matrices (self-attention or hybrid blocks); cross-attention does not fit the chain formulation.
- Computing lambda_2 eigenvalues is computationally heavy.
- Gains are larger for transformers with more structured attention maps, and the paper's ImageNet segmentation evaluation uses single-word prompts.
Experimental setup
Datasets: ImageNet segmentation benchmark ImageNet Imagenette COCO-Stuff-27 LAION-Aesthetics V2
Benchmarks: ImageNet segmentation COCO-Stuff-27 Imagenette linear probing ImageNet token masking
Metrics: Acc mIoU mAP IS FID KID mACC AUC
Baselines: LRP Partial-LRP Rollout ViT Attention GradCam DiffSeg TextSpan TransInterp DINO Attention DAAM FLUX Cross Attention Concept Attention
Builds on: PageRank Self-Attention Guidance DiffSeg ConceptAttention
Closest papers by meaning
- Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation (CVPR 2025 · similarity 0.88)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.87)
- Register and [CLS] tokens induce a decoupling of local and global features in large ViTs (NeurIPS 2025 · similarity 0.87)
- An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.87)
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.