Research map / Architectures and efficiency / Attention and transformers

ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers

CVPR 2024 · Architectures and efficiency › Attention and transformers · approach: Token merging and masked pretraining · 19 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ALGM, a parameter-free two-stage token merging method (local CLAP merging in layer 1, global bipartite merging mid-network, with unmerging) that speeds up plain-ViT semantic segmentation while often improving mIoU.

Problem

Token reduction for ViT-based semantic segmentation typically either hurts segmentation quality or gives limited efficiency gains, since every token needs a per-pixel prediction and existing merging methods designed for classification degrade mIoU.

Approach

Uses cosine similarity to identify mergeable tokens: local merging of similar tokens within small windows in the first layer (CLAP), followed by global bipartite merging (GBM, based on BSM/ToMe) halfway through the network, with an automatically derived threshold (mean + std of training similarities) and token unmerging before the decoder.

Key results

• On ADE20K, ALGM on Segmenter ViT-L raises mIoU from 51.8 to 52.7 and ALGM* gives up to +100% throughput (10 to 20 im/sec).
• ALGM on Segmenter ViT-S improves mIoU from 45.3 to 46.4 with throughput 134 to 192 im/sec and GFLOPs 38.6 to 26.3.
• ALGM* on Segmenter ViT-S reaches 45.5 mIoU with 235 im/sec, matching the baseline mIoU with about 46% fewer GFLOPs.

Measured effects reported in the paper

Surprising findings

• Token merging improves segmentation mIoU over the unreduced baseline on complex datasets, which the authors attribute to attention balancing and token denoising.
• Global cosine similarities in early layers do not reflect category correspondence, so global merging in layer 1 hurts mIoU.

Limitations

• On Cityscapes, visual homogeneity causes early tokens of different classes to be similar, requiring higher thresholds and limiting efficiency gains versus CTS.
• Comparisons with DToP and DoViT rely on relative differences because their code is unavailable and baseline numbers differ.
• Batched training requires taking the maximum remaining token count per batch, so adaptive reduction is partly constrained.

Experimental setup

Datasets: ADE20K Pascal Context Cityscapes COCO-Stuff-10K

Benchmarks: ADE20K Pascal Context Cityscapes COCO-Stuff-10K

Metrics: mIoU images per second GFLOPs

Baselines: CTS ToMe ELViT AiluRus DToP DoViT

Builds on: ToMe BSM CTS

Closest papers by meaning

• Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation (NeurIPS 2026 · similarity 0.87)
• Token Cropr: Faster ViTs for Quite a Few Tasks (CVPR 2025 · similarity 0.87)
• Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · similarity 0.86)
• DTMFormer: Dynamic Token Merging for Boosting Transformer-Based Medical Image Segmentation (AAAI 2024 · similarity 0.86)
• Subobject-level Image Tokenization (ICML 2025 · similarity 0.86)
• Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.86)