Research map / Detection and segmentation / CNN features
Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PRO-SCALE, a Mask2Former transformer encoder that progressively expands token length across layers (small-scale tokens first) plus a parameter-free Light Pixel Embedding, cutting encoder GFLOPs by about 52% on COCO with no PQ drop.
Problem
Mask2Former-style universal segmentation models spend over half their compute in the transformer encoder because every layer processes the full-length multi-scale token sequence, including many large-scale tokens.
Approach
Progressively expand the token sequence across encoder layers by introducing larger-scale features only in deeper layers (small-scale tokens first), and replace the convolutional per-pixel embedding with a parameter-free max-pooling Light Pixel Embedding module, with a Token Re-Calibration step to enrich small-scale tokens using the largest-scale features.
Key results
- With a SWIN-T backbone on COCO, PRO-SCALE (3,3,3) reaches 52.82 PQ at 171.7 GFLOPs versus Mask2Former's 52.03 PQ at 234.5 GFLOPs, a 26.8% total and 52% encoder reduction.
- With a SWIN-T backbone on Cityscapes, PRO-SCALE (3,3,3) reaches 63.06 PQ versus Lite-M2F's 62.29 PQ, with about 27% fewer GFLOPs than Mask2Former.
- PRO-SCALE (1,1,1) cuts encoder GFLOPs by about 80% on COCO with SWIN-T, at a cost of about 1.2 PQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace full-length encoder token processing with progressive token-length scaling | Mask2Former with SWIN-T · COCO val | PQ: 52.03 -> 52.82 | Helped |
| replace full-length encoder token processing with progressive token-length scaling | Mask2Former with SWIN-T · COCO val | total GFLOPs: 234.50 -> 171.70 | Helped |
| increase PRO scaling configuration from (1,1,1) to (3,3,3) (ablation) | PRO with SWIN-T · COCO val | PQ: 50.79 -> 52.82 | Helped |
| remove s1 (ablation) | M2F with SWIN-T · Cityscapes val | PQ: 64.00 -> 60.65 | Hurt |
| add s1 via LPE (ablation) | PRO (1,1,1) with SWIN-T · Cityscapes val | PQ: 60.58 -> 61.73 | Helped |
| replace max-pooling with average pooling in LPE (ablation) | PRO (3,3,3) with Res50 · Cityscapes | PQ: 61.87 -> 61.47 | Hurt |
| add LPE (ablation) | Lite-M2F with SWIN-T · COCO val | PQ: 52.70 -> 52.32 | Hurt |
| add LPE (ablation) | Lite-M2F with SWIN-T · Cityscapes val | PQ: 62.29 -> 61.35 | Hurt |
| set TRC temperature epsilon to 0.10 rather than 0.01 (ablation) | PRO with SWIN-T · Cityscapes | PQ: 60.05 -> 60.58 | Helped |
| increase TRC temperature epsilon from 0.10 to 1.00 (ablation) | PRO with SWIN-T · Cityscapes | PQ: 60.58 -> 60.32 | Hurt |
Surprising findings
- Simply removing the largest-scale feature from Mask2Former drops PQ by about 5.7 points on Cityscapes, while the LPE variant of the proposed method loses much less.
- Max-pooling in the Light Pixel Embedding outperforms average-pooling, and the parameter-free LPE performs close to the learned convolutional embedding.
Limitations
- Reported gains partly depend on the chosen (p1,p2,p3) configuration; more aggressive settings trade away PQ.
- Evaluation mostly on COCO and Cityscapes with lightweight backbones; larger backbones tested only on Cityscapes.
Experimental setup
Datasets: COCO Cityscapes ADE20K ImageNet-1K
Benchmarks: COCO Cityscapes ADE20K
Metrics: PQ AP mIoU GFLOPs FPS
Baselines: Mask2Former Lite-DETR RT-DETR PEM YOSO ReMaX RAP-SAM MaskFormer
Builds on: Mask2Former Deformable DETR MaskFormer DETR
Closest papers by meaning
- Scaling White-Box Transformers for Vision (NeurIPS 2024 · similarity 0.86)
- PEM: Prototype-based Efficient MaskFormer for Image Segmentation (CVPR 2024 · similarity 0.85)
- Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More (ICML 2025 · similarity 0.85)
- Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers (ICML 2025 · similarity 0.84)
- EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (CVPR 2024 · similarity 0.84)
- ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generation (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.