Research map / Architectures and efficiency / Attention and transformers
Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LongMIL, a local-global hybrid Transformer for whole slide image MIL that uses a 2-D local attention mask (computed chunk-wise) plus a final global attention layer to overcome the low-rank bottleneck and quadratic cost of full self-attention.
Problem
Transformer-based multiple instance learning for gigapixel histopathology WSIs suffers from a low-rank attention bottleneck (sequence length far exceeds embedding size), quadratic cost, and poor extrapolation to shape-varying slides, while prior fixes (region slicing in HIPT, Nystromformer in TransM
Approach
Argue that the attention matrix's rank is bounded by the embedding size for long WSI sequences, and show that a local band attention mask raises the rank (lower bound n-b) and reduces cost to linear via chunked computation; combine two local-attention layers with a 2x2 pooling and one full global attention layer, using 2-D Euclidean locality rather than 1-D sequence locality.
Key results
- On BRACS tumor subtyping with Lunit ViT-S features, LongMIL reaches F1 0.706 and AUC 0.888, versus 0.689/0.870 for full attention and 0.648/0.835 for TransMIL.
- On survival prediction, LongMIL achieves C-index 0.624 on COADREAD, 0.589 on STAD and 0.619 on BRCA, the best among the compared methods.
- On BRACS with UNI features at 224 patch size, LongMIL reaches F1 0.728 and AUC 0.887, outperforming full attention (0.715/0.884).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LongMIL instead of Full Attention | WSI tumor subtyping · BRACS (ViT-S Lunit) | F1: 0.689 0.036 -> 0.706 0.025 | Helped |
| use LongMIL instead of Full Attention | WSI tumor subtyping · BRACS (ViT-S DINO) | F1: 0.648 0.028 -> 0.657 0.026 | Helped |
| use LongMIL instead of Full Attention | WSI survival prediction · COADREAD | C-index: 0.603 0.048 -> 0.624 0.057 | Helped |
| use LongMIL instead of Full Attention | WSI tumor subtyping · BRACS (UNI) | F1: 0.715 0.043 -> 0.728 0.045 | Helped |
| use LongMIL instead of Full Attention | WSI tumor subtyping · BRACS (GigaPath) | F1: 0.663 0.023 -> 0.673 0.023 | Hurt |
| replace global full attention with linear attention (ablation) | LongMIL · BRACS tumor subtyping (20x) | F1: 0.706 0.025 -> 0.693 0.024 | Hurt |
| replace global full attention with linear attention (ablation) | LongMIL · BRACS tumor subtyping (40x) | F1: 0.624 0.060 -> 0.622 0.055 | Hurt |
| increase patch size from 224 to 448 (ablation) | LongMIL with UNI features · BRACS tumor subtyping | F1: 0.728 0.05 -> 0.722 0.04 | Hurt |
| replace global full attention with V-Mamba (ablation) | LongMIL · BRACS tumor subtyping | F1: 0.706 0.025 -> 0.693 0.024 | Hurt |
| train on small WSIs and test on larger unseen WSIs using LongMIL | WSI tumor subtyping · BRACS extrapolation | performance | Helped |
Surprising findings
- Full self-attention with absolute/relative positions is often outperformed by vanilla attention MIL methods, and TransMIL (Nystromformer approximation) underperforms even simple baselines on several tasks.
- Linear attention and linear RNN variants (RetNet, GLA, Mamba) underperform full attention as standalone MIL models, and combining them with local attention did not reliably beat the full-attention hybrid.
Limitations
- Restricted to very large images since the method relies on Transformer modelling over long sequences.
- Uses limited embedding size for fair comparison, while the low-rank argument suggests larger embeddings would help performance.
- Gains on several tasks are small relative to standard deviations, and extrapolation improvement is only marginally significant (p-value near 0.1).
Experimental setup
Datasets: BRACS TCGA-BRCA TCGA-COADREAD TCGA-STAD
Benchmarks: BRACS TCGA-BRCA TCGA-COADREAD TCGA-STAD
Metrics: macro-F1 macro-AUC C-index
Baselines: AB-MIL DS-MIL CLAM-SB DTFD-MIL TransMIL HIPT Full Attention GigaPath WSI head
Builds on: TransMIL HIPT FlashAttention RoPE ALiBi
Closest papers by meaning
- Dynamic Graph Representation with Knowledge-aware Attention for Histopathology Whole Slide Image Analysis (CVPR 2024 · similarity 0.91)
- Transformer-Based Video-Structure Multi-Instance Learning for Whole Slide Image Classification (AAAI 2024 · similarity 0.90)
- Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images (ICML 2025 · similarity 0.89)
- Sequential Attention-based Sampling for Histopathological Analysis (NeurIPS 2025 · similarity 0.88)
- Nonparametric Distribution Matching for Self-Supervised Whole-Slide Image Condensation (NeurIPS 2026 · similarity 0.88)
- PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.