Research map / Detection and segmentation / CNN features
PEM: Prototype-based Efficient MaskFormer for Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PEM, an efficient MaskFormer-style multi-task segmentation architecture with prototype-based masked cross-attention and a fully convolutional FPN using context-based self-modulation and deformable convolutions.
Problem
Transformer-based mask classification segmentation models (MaskFormer, Mask2Former) are computationally expensive, especially in cross-attention over high-resolution features, limiting deployment on edge devices.
Approach
Select one prototype pixel per object query via similarity argmax, then replace dot-product cross-attention with cheap element-wise query-prototype interactions; pair this with a convolutional FPN that restores global context (CSM) and deformability (deformable convs) instead of deformable attention.
Key results
- PEM reaches 61.1 PQ on Cityscapes panoptic at 13.8 FPS, the fastest among compared multi-task architectures, with Mask2Former at 62.1 PQ but 4.1 FPS.
- PEM attains 38.5 PQ on ADE20K panoptic at 35.7 FPS, outperforming YOSO (38.0 PQ) and surpassing it in speed-accuracy trade-off.
- PEM reaches 45.5 mIoU on ADE20K semantic segmentation at 35.7 FPS, outperforming task-specific PIDNet-L (40.5 mIoU) by about 5 mIoU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PEM instead of YOSO | ResNet50 segmentation models · Cityscapes panoptic segmentation | PQ: 59.7 -> 61.1 | Helped |
| use PEM instead of Mask2Former | ResNet50 segmentation models · ADE20K panoptic segmentation | PQ: 39.7 -> 38.5 | Hurt |
| use PEM instead of YOSO | ResNet50 segmentation models · Cityscapes semantic segmentation | mIoU: 79.4 -> 79.9 | Helped |
| use PEM instead of PIDNet-L | semantic segmentation models · ADE20K semantic segmentation | mIoU: 40.5 -> 45.5 | Helped |
| add prototype selection (ablation) | PEM-CA · Cityscapes panoptic segmentation | PQ: 48.7 -> 61.1 | Helped |
| add the foreground masking mechanism (ablation) | PEM-CA · Cityscapes panoptic segmentation | PQ: 57.8 -> 61.1 | Helped |
| replace PEM-CA with standard cross-attention (ablation) | PEM · Cityscapes panoptic segmentation | PQ: 58.4 -> 61.1 | Hurt |
| replace PEM-CA with masked cross-attention (ablation) | PEM · Cityscapes panoptic segmentation | PQ: 60.4 -> 61.1 | Hurt |
| replace the MaskFormer pixel decoder with the PEM pixel decoder (ablation) | PEM · Cityscapes panoptic segmentation | PQ: 57.8 -> 61.1 | Helped |
| replace the Mask2Former pixel decoder with the PEM pixel decoder (ablation) | PEM · Cityscapes panoptic segmentation | PQ: 61.4 -> 61.1 | Hurt |
| add context-based self-modulation modules (ablation) | PEM pixel decoder · Cityscapes panoptic segmentation | PQ: 60.0 -> 61.1 | Helped |
| use deformable convolutions instead of standard convolutions (ablation) | PEM pixel decoder · Cityscapes panoptic segmentation | PQ: 57.1 -> 61.1 | Helped |
Surprising findings
- Removing prototype selection (token aggregation by summation) drops Cityscapes PQ by 13.4, showing tokens cannot be naively collapsed.
- On ADE20K the model attains nearly zero PQ without cross-attention layers, showing query refinement is essential.
Limitations
- Gains over Mask2Former are not uniform; Mask2Former remains more accurate on both datasets.
- On Cityscapes semantic segmentation PIDNet outperforms PEM, so task-specific architectures are not always beaten.
- Evaluation is limited to two datasets and semantic/panoptic tasks.
Experimental setup
Datasets: Cityscapes ADE20K ImageNet-1k
Benchmarks: Cityscapes ADE20K
Metrics: PQ PQth PQst mIoU FPS FLOPs Params latency
Baselines: Mask2Former MaskFormer YOSO kMaxDeepLab PIDNet UPSNet PanDeepLab STDC
Builds on: MaskFormer Mask2Former DETR SwiftFormer
Closest papers by meaning
- Masked-Attention Mask Transformer for Universal Image Segmentation (CVPR 2022 · similarity 0.90)
- Per-Pixel Classification is Not All You Need for Semantic Segmentation (NeurIPS 2021 · similarity 0.88)
- MaskDiff: Modeling Mask Distribution with Diffusion Probabilistic Model for Few-Shot Instance Segmentation (AAAI 2024 · similarity 0.85)
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.85)
- Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation (ICLR 2025 · similarity 0.85)
- Your ViT is Secretly an Image Segmentation Model (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.