Research map / Detection and segmentation / Weakly supervised detection and segmentation
FFR: Frequency Feature Rectification for Weakly Supervised Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Frequency Feature Rectification (FFR), a single-stage WSSS framework that uses frequency masking and prototype-guided token selection to rectify over-smoothed false segmentations caused by attenuated high-frequency ViT features, plus a dual-branch decoder for boundary detail.
Problem
Single-stage ViT-based weakly supervised semantic segmentation from image-level labels suffers from over-smoothed false segmentations along object boundaries, which the paper attributes to attenuated high-frequency features in ViT.
Approach
Use DFT-based low-pass masking of decoder patch tokens to obtain reliable low-frequency features, select tokens whose high-frequency features disagree with prototype-based low-frequency correlation, and rectify them with a cross-entropy loss; add a dual-branch segmentation head with a low-dilation branch for high-frequency features.
Key results
- Achieves 76.0% mIoU on PASCAL VOC 2012 val and 75.5% on test (with dense CRF), new single-stage state of the art, outperforming PFSR by 0.3% and 0.5%.
- Achieves 76.4% pseudo-label mIoU on VOC val, 2.3% above DuPL.
- Reaches 46.8% mIoU on MS COCO 2014 val, slightly above SeCo (46.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply FFR instead of PFSR | ViT-B WSSS · PASCAL VOC 2012 val | mIoU: 75.7 -> 76.0 | Helped |
| Apply FFR instead of PFSR | ViT-B WSSS · PASCAL VOC 2012 test | mIoU: 75.0 -> 75.5 | Helped |
| Apply FFR instead of SeCo | ViT-B WSSS · MS COCO 2014 val | mIoU: 46.7 -> 46.8 | Helped |
| Add the feature rectification loss without mask Bl (ablation) | FFR · PASCAL VOC 2012 val | mIoU (segmentation): 72.1 -> 70.4 | Hurt |
| Add the feature rectification loss with false-token mask Bl (ablation) | FFR · PASCAL VOC 2012 val | mIoU (segmentation): 72.1 -> 73.5 | Helped |
| Add high-frequency enhanced segmentation loss Lhs (ablation) | FFR · PASCAL VOC 2012 val | mIoU (segmentation): 72.1 -> 74.1 | Helped |
| Add Lr with mask and Lhs together (ablation) | FFR · PASCAL VOC 2012 val | mIoU (segmentation): 72.1 -> 74.8 | Helped |
| Set FSTS threshold α to 0.1 (ablation) | FFR · PASCAL VOC 2012 val | mIoU (segmentation): 74.8 -> 72.8 | Hurt |
| Set FSTS threshold α to 0.2 (ablation) | FFR · PASCAL VOC 2012 val | mIoU (segmentation): 74.8 -> 73.4 | Hurt |
| Set FSTS threshold α to 0.4 (ablation) | FFR · PASCAL VOC 2012 val | mIoU (segmentation): 74.8 -> 74.1 | Hurt |
| Set FSTS threshold α to 0.5 (ablation) | FFR · PASCAL VOC 2012 val | mIoU (segmentation): 74.8 -> 74.0 | Hurt |
Surprising findings
- Adding the rectification loss without the frequency mask lowers segmentation mIoU by 1.7% (70.4 vs 72.1 baseline), showing indiscriminate low-frequency transfer harms performance.
- Segmentation from low-frequency components alone can exceed segmentation from original features at certain low-frequency ranges.
Limitations
- Gains over the strongest single-stage competitor are small (0.3% on VOC val, 0.1% on COCO val vs SeCo).
- Relies on hyperparameters (mu, alpha, lambda) tuned by ablation on the validation set.
- High-frequency features alone produce poor segmentation, so the method depends on low-frequency guidance.
Experimental setup
Datasets: PASCAL VOC 2012 SBD MS COCO 2014
Benchmarks: PASCAL VOC 2012 MS COCO 2014
Metrics: mIoU
Baselines: ToCo PFSR DuPL SeCo DALNet AFA CLIP-ES KTSE
Builds on: ToCo DeepLab-LargeFOV dense CRF
Closest papers by meaning
- When Semantic Segmentation Meets Frequency Aliasing (ICLR 2024 · similarity 0.86)
- Frequency-Adaptive Dilated Convolution for Semantic Segmentation (CVPR 2024 · similarity 0.86)
- Lightweight Frequency Masker for Cross-Domain Few-Shot Semantic Segmentation (NeurIPS 2024 · similarity 0.85)
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.85)
- SFC: Shared Feature Calibration in Weakly Supervised Semantic Segmentation (AAAI 2024 · similarity 0.84)
- Progressive Feature Self-Reinforcement for Weakly Supervised Semantic Segmentation (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.