Research map / Detection and segmentation / Weakly supervised detection and segmentation
Class Tokens Infusion for Weakly Supervised Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Class Token Infusion (intra-image and cross-image) plus a background class token in a DeiT-S multi-class-token ViT to produce better CAMs for image-level weakly supervised semantic segmentation.
Problem
Multi-class-token ViT-based WSSS produces CAMs whose class tokens are highly correlated across images and over-expand into false-positive background regions.
Approach
Infuse class tokens between augmented views of the same image (I-CTI) and between images sharing a class (C-CTI) at an intermediate layer, enforcing CAM consistency, and add a learnable background class token trained with a background CAM loss to suppress false activations.
Key results
- Achieves 69.5% mIoU on VOC 2012 train CAM seeds (vs 64.7% baseline) and 73.7% pseudo-mask mIoU, with the segmentation model reaching 74.1% val and 73.2% test.
- On MS COCO 2014 val, achieves 45.4% mIoU, above ACR (45.3%) and USAGE (44.3%).
- Ablation: background token alone gives +2.9 points; I-CTI adds +1.1; C-CTI brings the total to 69.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the background token and background CAM loss (ablation) | ViT-based WSSS baseline · PASCAL VOC 2012 train | mIoU: 64.7 -> 67.6 | Helped |
| Add I-CTI to the background CAM loss setup (ablation) | ViT-based WSSS · PASCAL VOC 2012 train | mIoU: 67.6 -> 68.7 | Helped |
| Add C-CTI to the background CAM loss setup (ablation) | ViT-based WSSS · PASCAL VOC 2012 train | mIoU: 67.6 -> 69.0 | Helped |
| Add both I-CTI and C-CTI (ablation) | ViT-based WSSS with background CAM loss · PASCAL VOC 2012 train | mIoU: 67.6 -> 69.5 | Helped |
| Remove the background class token while retaining background CAM training (ablation) | ViT-based WSSS · PASCAL VOC 2012 train | mIoU: 4.1%p decrease compared to the baseline | Hurt |
| Set the CTI infusion index to different transformer layers (ablation) | Full CTI · PASCAL VOC 2012 train | mIoU: 68.8, 69.5, 68.6, 68.9, 69.4, 68.7, 69.0, 68.7, 68.8 | Mixed |
| Use the proposed CTI framework | PASCAL VOC 2012 semantic segmentation · PASCAL VOC 2012 val | mIoU: 72.0 -> 74.1 | Helped |
| Use the proposed CTI framework | PASCAL VOC 2012 semantic segmentation · PASCAL VOC 2012 test | mIoU: 73.0 -> 73.2 | Helped |
| Use the proposed CTI framework | MS COCO 2014 semantic segmentation · MS COCO 2014 val | mIoU: 45.3 -> 45.4 | Helped |
| Use the proposed CTI framework | PASCAL VOC 2012 CAM generation · PASCAL VOC 2012 train CAMs | Seed mIoU: 67.7 -> 69.5 | Helped |
| Use the proposed CTI framework | PASCAL VOC 2012 pseudo-label generation · PASCAL VOC 2012 train pseudo-ground-truth | Mask mIoU: 72.8 -> 73.7 | Helped |
Surprising findings
- Using a background CAM loss without the background class token drops mIoU by 4.1 points relative to the baseline.
Limitations
- Relies on IRN post-processing for pseudo-label generation, so improvements are tied to CAM seed quality.
- Gains on MS COCO remain modest relative to VOC, and ViT-based methods still lag on COCO false activations.
Experimental setup
Datasets: PASCAL VOC 2012 MS COCO 2014
Benchmarks: PASCAL VOC 2012 MS COCO 2014
Metrics: mIoU precision recall
Baselines: MCTformer ACR USAGE FPR AEFT OC-CSE ReCAM RIB
Builds on: MCTformer DeiT ToCo IRNet
Closest papers by meaning
- Weakly Supervised Semantic Segmentation via Progressive Confidence Region Expansion (CVPR 2025 · similarity 0.89)
- From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.88)
- MoRe: Class Patch Attention Needs Regularization for Weakly Supervised Semantic Segmentation (AAAI 2025 · similarity 0.88)
- Multi-Label Prototype Visual Spatial Search for Weakly Supervised Semantic Segmentation (CVPR 2025 · similarity 0.87)
- Progressive Feature Self-Reinforcement for Weakly Supervised Semantic Segmentation (AAAI 2024 · similarity 0.87)
- Separate and Conquer: Decoupling Co-occurrence via Decomposition and Representation for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.