Research map / Video / Video segmentation and tracking
Transformer-Based Video-Structure Multi-Instance Learning for Whole Slide Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Models a whole slide image as a synthetic pathologist-observation video of spatially consecutive patch clips processed by a shared Transformer with class-specific clip tokens, enabling end-to-end WSI classification and instance-level localization.
Problem
Whole slide image classification is hampered by gigapixel size: two-stage MIL methods use frozen pretrained extractors (domain gap, inductive bias) and cannot be trained end-to-end, while end-to-end methods often ignore contextual relations between patches.
Approach
Group spatially adjacent patches into synthetic video clips, process each clip with a parameter-shared Transformer (MHSA) with class-specific clip tokens trained end-to-end with a ResNet18 encoder, and derive instance pseudo-labels from dot products between clip tokens and patch features.
Key results
- VINO-E2E reaches 0.9466 bag-level AUC and 0.9213 instance-level AUC on CAMELYON16, above the compared methods.
- On PANDA, VINO-E2E achieves the best bag and instance AUCs among end-to-end and two-stage methods for Grades 3-5, e.g., 0.9173 bag AUC for Grade 3.
- On TCGA-NSCLC, VINO-Feature reaches 0.9834 bag AUC per text, though the Table 3 value is 0.9853, and outperforms the other methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VINO-E2E instead of Loss-Attention | CAMELYON16 instance classification · CAMELYON16 | instance-level AUC: 0.8995 -> 0.9213 | Helped |
| use VINO-E2E instead of IBMIL | CAMELYON16 bag classification · CAMELYON16 | bag-level AUC: 0.8991 -> 0.9466 | Helped |
| use VINO-E2E instead of DSMIL | PANDA instance classification · PANDA Grade 3 | instance-level AUC: 0.9525 -> 0.9659 | Helped |
| use VINO-E2E instead of DSMIL | PANDA instance classification · PANDA Grade 4 | instance-level AUC: 0.9363 -> 0.9406 | Helped |
| use VINO-E2E instead of IBMIL | PANDA bag classification · PANDA Grade 3 | bag-level AUC: 0.9015 -> 0.9173 | Helped |
| use VINO-E2E instead of IBMIL | PANDA bag classification · PANDA Grade 4 | bag-level AUC: 0.8923 -> 0.9048 | Helped |
| use VINO-E2E instead of IBMIL | PANDA bag classification · PANDA Grade 5 | bag-level AUC: 0.9010 -> 0.9140 | Helped |
| use VINO-Feature instead of TransMIL | TCGA-NSCLC bag classification · TCGA-NSCLC | bag-level AUC: 0.9830 -> 0.9853 | Helped |
| preserve neighboring-patch context when constructing video clips (ablation) | VINO · CAMELYON16 | instance-level AUC: 0.8925 -> 0.9213 | Helped |
| preserve neighboring-patch context when constructing video clips (ablation) | VINO · CAMELYON16 | bag-level AUC: 0.9276 -> 0.9466 | Helped |
| train end-to-end instead of using pre-extracted features (ablation) | VINO · CAMELYON16 | instance-level AUC: 0.9185 -> 0.9213 | Helped |
| train end-to-end instead of using pre-extracted features (ablation) | VINO · CAMELYON16 | bag-level AUC: 0.9085 -> 0.9466 | Helped |
Surprising findings
- Traditional video anomaly detection (RTFM) performs poorly on WSI video clips (0.7500 bag AUC on CAMELYON16).
Limitations
- Baselines for end-to-end comparison sampled only 64 instances per batch due to memory constraints.
- Clip construction assumes position corresponds to temporal order, a simplification that is not a true pathologist observation process.
Experimental setup
Datasets: CAMELYON16 PANDA TCGA-NSCLC
Benchmarks: CAMELYON16 PANDA TCGA-NSCLC
Metrics: bag-level AUC instance-level AUC
Baselines: ABMIL Loss-Attention DSMIL TransMIL DTFD-MIL IBMIL Chikontwe et al. top-k instance method Loss-ABMIL RTFM
Builds on: TransMIL MCT-Former ABMIL DSMIL RTFM
Closest papers by meaning
- Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis (NeurIPS 2024 · similarity 0.90)
- CaMIL: Causal Multiple Instance Learning for Whole Slide Image Classification (AAAI 2024 · similarity 0.88)
- PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization (NeurIPS 2025 · similarity 0.88)
- Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification (CVPR 2024 · similarity 0.87)
- Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction (CVPR 2024 · similarity 0.87)
- Queryable Prototype Multiple Instance Learning with Vision-Language Models for Incremental Whole Slide Image Classification (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.