Research map / Detection and segmentation / Weakly supervised detection and segmentation
Vision Transformers Are Good Mask Auto-Labelers
Shiyi Lan, Xitong Yang, Zhiding Yu, Zuxuan Wu, Jose M. Alvarez, Anima Anandkumar
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use attention-based mask decoder instead of fully connected or fully convolutional decoder in phase 1 (ablation) | MAL mask decoder · COCO val2017 | Mask AP: 42.3 vs 35.5 (FC) and 36.1 (FCN) | Improved |
| use Mask2Former-style query-based decoder in phase 1 (ablation) | MAL mask decoder · COCO val2017 | Mask AP: no reasonable results obtained | Hurt |
| use MAE pretraining for ViT encoder instead of DeiT or Swin/ConvNeXt encoders (ablation) | MAL image encoder · COCO val2017 | Mask AP: 42.3 (MAE-Base) vs 41.1 (DeiT-Base), 40.8 (DeiT-Small), 40.2 (Swin-Base), 39.6 (ConvNeXt-Base) | Improved |
| scale image encoder from ViT-MAE-Base to ViT-MAE-Large (ablation) | MAL image encoder · COCO val2017 | Mask AP: 42.3 for both Base and Large | No clear effect |
| box expansion upper bound theta increased from 0.6 to 1.2 (ablation) | MAL RoI input sampling · COCO val2017 | Mask AP: 41.3 at theta 0.6 to 42.3 at theta 1.2 | Improved |
| increase box expansion upper bound theta beyond 1.2 to 1.6 (ablation) | MAL RoI input sampling · COCO val2017 | Mask AP: 42.0 at 1.4 and 41.8 at 1.6 vs 42.3 at 1.2 | Hurt |
| use box expansion RoI sampling with background bags rather than pure box cropping (ablation) | MAL RoI input generation · COCO | qualitative mask quality | Improved |
| use MAL-generated masks to train Mask2Former (Swin-Small) instead of GT masks | two-phase box-supervised pipeline · COCO test-dev 2017 | Mask AP: 44.1 test-dev vs 40.0 for BoxTeacher box-supervised | Improved |
| train MAL on COCO and generate LVIS pseudo-labels (open-vocabulary) instead of training on LVIS | MAL labeler · LVIS v1 val | Mask AP: about 0.35% mAP drop vs LVIS-trained (23.3 vs 23.7 with X101-32x4d, 24.2 vs 24.5 with X101-64x4d) | No clear effect |
Closest papers by meaning
- SIM: Semantic-Aware Instance Mask Generation for Box-Supervised Instance Segmentation (CVPR 2023 · similarity 0.90)
- BoxInst: High-Performance Instance Segmentation With Box Annotations (CVPR 2021 · similarity 0.90)
- BoxTeacher: Exploring High-Quality Pseudo Labels for Weakly Supervised Instance Segmentation (CVPR 2023 · similarity 0.89)
- Per-Pixel Classification is Not All You Need for Semantic Segmentation (NeurIPS 2021 · similarity 0.88)
- MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving (AAAI 2024 · similarity 0.87)
- Extreme Point Supervised Instance Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
