Research map / Detection and segmentation / Weakly supervised detection and segmentation
Weakly Supervised Semantic Segmentation for Driving Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a CLIP-based WSSS framework for driving scenes that combines global-local view training (local patch sampling with MaskCLIP pseudo-masks) and consistency-aware region balancing (CARB) to reweight noisy pseudo-mask regions, reaching 51.8% mIoU on Cityscapes test.
Problem
Image-level weakly supervised semantic segmentation degrades severely on driving scene datasets such as Cityscapes, due to many co-occurring classes, few negative samples, and small objects, and CLIP-derived pseudo-masks miss small objects and are noisy.
Approach
Exploit CLIP's scale-dependent pseudo-masks by training on both global images and small local patches, and weight the loss of pseudo-mask pixels by whether they agree with the segmentation network's own predictions, using an adaptive ratio of consistent-to-inconsistent region losses instead of discarding inconsistent pixels.
Key results
- Achieves 51.8% mIoU on Cityscapes test, outperforming CLIP-ES (35.0%) by 16.8 points in the reported comparison.
- Reaches 50.5% mIoU on CamVid test versus 39.6% for CLIP-ES, and 32.2% mIoU on WildDash2 val versus 24.7% for CLIP-ES.
- Ablation on Cityscapes val: base 40.1, +CARB 45.7, +Local 45.1, +Local+CARB 50.6, +Dual+CARB 52.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CARB to the base method (ablation) | Base · Cityscapes val | mIoU: 40.1 -> 45.7 | Helped |
| Add Local view sampling to the base method (ablation) | Base · Cityscapes val | mIoU: 40.1 -> 45.1 | Helped |
| Add Local view sampling and CARB (ablation) | Base · Cityscapes val | mIoU: 40.1 -> 50.6 | Helped |
| Use Dual mask recreation instead of Local view sampling (ablation) | Base · Cityscapes val | mIoU: 45.1 -> 45.8 | Helped |
| Add Dual mask recreation and CARB (ablation) | Base · Cityscapes val | mIoU: 40.1 -> 52.1 | Helped |
| Use Dual mask recreation with CARB instead of Local view sampling with CARB (ablation) | Base + CARB · Cityscapes val | mIoU: 50.6 -> 52.1 | Helped |
| Set fixed inconsistent-region loss weight to 0.1 instead of adaptive region balancing (ablation) | CARB · Cityscapes val | mIoU: 52.1 -> 51.06 | Hurt |
| Use a large resize rate (ablation) | Local view sampling · Cityscapes | classwise mIoU | Mixed |
| Use resize ratios other than 1.0 (ablation) | Local view sampling · Cityscapes val | mIoU: 52.1% | Hurt |
| Use 256×256 local patches instead of 512×512 patches (ablation) | Local view sampling · Cityscapes val | mIoU | Hurt |
| Use adaptive region balancing instead of a fixed loss weight (ablation) | Region balancing · Cityscapes val | mIoU: 51.06% -> 52.1% | Helped |
| Use the proposed method instead of CLIP-ES | WSSS segmentation · Cityscapes test | mIoU: 35.0 -> 51.8 | Helped |
Surprising findings
- CAM-based WSSS methods such as AMN and CLIMS perform very poorly on driving scenes (CLIMS reaches 1.0% on WildDash2), and some CAM-based methods fall below random guessing.
- CamVid's road and car classes appear in all training images, so they cannot be separated using image-level labels alone; CLIP's open-vocabulary ability is used to distinguish them.
Limitations
- Results compared against methods designed for object-centric datasets, so comparisons may be unfair as the authors acknowledge.
- Fixed-weight alternative performs near the adaptive method but requires hyperparameter search; the adaptive method's gains depend on the loss ratio heuristic.
- Local view crop size and resize ratio are tuned empirically, with performance sensitive to these choices.
Experimental setup
Datasets: Cityscapes CamVid WildDash2 PASCAL VOC MS COCO
Benchmarks: Cityscapes CamVid WildDash2
Metrics: mIoU
Baselines: AffinityNet SEAM 1-Stage SEC CAM AMN CLIMS CLIP-ES
Builds on: MaskCLIP CLIP L2G CLIP-ES
Closest papers by meaning
- Progressive Feature Self-Reinforcement for Weakly Supervised Semantic Segmentation (AAAI 2024 · similarity 0.89)
- Exploring Scene Affinity for Semi-Supervised LiDAR Semantic Segmentation (CVPR 2025 · similarity 0.88)
- MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving (AAAI 2024 · similarity 0.88)
- Weakly Supervised Semantic Segmentation via Progressive Confidence Region Expansion (CVPR 2025 · similarity 0.87)
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.87)
- Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.