Research map / Architectures and efficiency / Attention and transformers
DTMFormer: Dynamic Token Merging for Boosting Transformer-Based Medical Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DTMFormer, a plug-and-play transformer block that dynamically merges tokens into fewer semantic tokens (attention-guided clustering) to counter attention collapse and improve convergence in medical image segmentation.
Problem
Transformers in CNN-Transformer hybrid medical image segmentation suffer from attention collapse (uniform dependency distributions), attributed to O(n^2) complexity, limited annotated data, and redundant token dependencies.
Approach
An attention-guided token merging (ATM) module selects semantic cluster centers using attention-based dependence scores plus DPC-KNN feature density scores, merges tokens into them, then runs self-attention over fewer tokens; a light token reconstruction (LTR) module restores resolution and fuses ordinary and semantic tokens. The goal is performance gain rather than only compression.
Key results
- On ACDC, SETR+DTMFormer improves average Dice from 88.56 to 89.64, and FATNet+DTMFormer reaches 92.53 average Dice, outperforming D-Former (92.29).
- On ISIC, FATNet+DTMFormer improves Dice from 89.05 to 90.40 and reduces HD from 6.73 to 4.99.
- On SETR, DTMFormer cuts GFLOPs from 53.2 to 9.8 and parameters from 51.5M to 33.5M while improving Dice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DTMFormer to SETR | SETR · ACDC | Dice: 88.56 -> 89.64 | Helped |
| add DTMFormer to TransFuse | TransFuse · ACDC | Dice: 90.48 -> 90.95 | Helped |
| add DTMFormer to TransUnet | TransUnet · ACDC | Dice: 91.83 -> 92.30 | Helped |
| add DTMFormer to FATNet | FATNet · ACDC | Dice: 91.94 -> 92.53 | Helped |
| add DTMFormer to SETR | SETR · ISIC | Dice: 88.42 -> 88.98 | Helped |
| add DTMFormer to TransFuse | TransFuse · ISIC | Dice: 88.00 -> 89.21 | Helped |
| add DTMFormer to TransUnet | TransUnet · ISIC | Dice: 89.19 -> 89.67 | Helped |
| add DTMFormer to FATNet | FATNet · ISIC | Dice: 89.05 -> 90.40 | Helped |
| add DTMFormer to SETR | SETR · BTCV | Dice: 75.23 -> 79.31 | Helped |
| add DTMFormer to TransFuse | TransFuse · BTCV | Dice: 71.84 -> 72.56 | Helped |
| add DTMFormer to TransUnet | TransUnet · BTCV | Dice: 81.99 -> 82.27 | Helped |
| add DTMFormer to FATNet | FATNet · BTCV | Dice: 84.44 -> 85.25 | Helped |
Surprising findings
- Token pruning/merging methods like DViT and CTM either degrade segmentation or fail to improve it, while the proposed merging improves Dice and reduces complexity simultaneously.
- Multi-stage token merging was found to hurt performance in medical segmentation because over-abstracted tokens cause attention collapse.
Limitations
- Gains are modest on some baselines (e.g., 0.47%-1.08% Dice on ACDC), and some configurations regress (e.g., SETR HD on ISIC worsens).
- Only 2D evaluation; 3D settings are not explored.
- Performance degrades with large sparsity ratio r or large k, requiring tuning.
Experimental setup
Datasets: ACDC ISIC BTCV
Benchmarks: ACDC ISIC BTCV
Metrics: Dice HD IoU Accuracy parameters GFLOPs
Baselines: SETR TransFuse TransUNet FATNet DViT CTM APFormer D-Former nnFormer MISSFormer
Builds on: DPC-KNN TCFormer DViT TransUNet FATNet
Closest papers by meaning
- TMFormer: Token Merging Transformer for Brain Tumor Segmentation with Missing Modalities (AAAI 2024 · similarity 0.88)
- DDViT: Double-Level Fusion Domain Adapter Vision Transformer (Student Abstract) (AAAI 2024 · similarity 0.87)
- Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · similarity 0.87)
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.86)
- ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers (CVPR 2024 · similarity 0.86)
- nnWNet: Rethinking the Use of Transformers in Biomedical Image Segmentation and Calling for a Unified Evaluation Benchmark (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.