Research map / Vision-language and multimodal / CLIP and zero-shot
LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free open-vocabulary semantic segmentation that refines VLM (MaskCLIP) patch predictions with label propagation over DINO patch graphs across all sliding windows, then pixel-level LP (LPOSS+).
Problem
Open-vocabulary semantic segmentation with VLMs yields coarse patch-level predictions and window-based inference that limits boundary accuracy and cross-window context.
Approach
Applies classical label propagation (quadratic cost with unary and binary terms) over a joint patch graph built from DINO features spanning all sliding windows, decoupling feature extraction from prediction, and repeats LP at the pixel level (Lab color graph) as a boundary refinement step.
Key results
- LPOSS+ achieves average mIoU of 42.1 over eight datasets with ViT-B/16 (Table 1), state-of-the-art among training-free methods, versus 41.1 for ProxyCLIP.
- LPOSS+ reaches average Boundary IoU of 32.1, versus 29.0 for ProxyCLIP (Table 2).
- Ensembling two window setups raises LPOSS+ average mIoU to 42.9 (Table 3); LPOSS+ also improves DINOv2 linear-head predictions on ADE20k from 47.3% to 49.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add pixel-level label propagation to patch-level LPOSS (ablation) | LPOSS · Eight-dataset average | mIoU: 41.3 -> 42.1 | Helped |
| Add pixel-level label propagation to patch-level LPOSS (ablation) | LPOSS · Eight-dataset average | Boundary IoU: 30.3 -> 32.1 | Helped |
| Use LPOSS+ rather than the strongest reported baseline | Open-vocabulary segmentation · Eight-dataset average | mIoU: 41.1 -> 42.1 | Helped |
| Use LPOSS+ rather than the strongest reported baseline | Open-vocabulary segmentation · Eight-dataset average | Boundary IoU: 29.0 -> 32.1 | Helped |
| Use no sliding windows for feature extraction (ablation) | LPOSS+ · Eight-dataset average | mIoU: 42.1 -> 38.9 | Hurt |
| Increase window size and stride from 224 × 224 / 112 × 112 to 448 × 448 / 224 × 224 (ablation) | LPOSS+ · Eight-dataset average | mIoU: 42.1 -> 40.8 | Hurt |
| Ensemble the two window-size setups (ablation) | LPOSS+ · Eight-dataset average | mIoU: 42.1 -> 42.9 | Helped |
| Apply LPOSS+ refinement to DINOv2 linear segmentation predictions | DINOv2 linear segmentation head · ADE20k | mIoU: 47.3% -> 49.5% | Helped |
Surprising findings
- Ground-truth downsampled to patch resolution and upsampled still achieves about 85% mIoU on average, showing patch-level prediction itself limits accuracy.
- Smaller sliding windows (224) work better for LPOSS than the common 448 setting, unlike other methods that prefer larger windows.
Limitations
- Performance on VOC20 is notably lower than some competitors due to window-size choice.
- ProxyCLIP significantly outperforms LPOSS+ on the Object dataset, attributed to its dataset-specific background thresholds.
- Joint feature extraction across the whole image degrades performance, so window-based feature extraction is still needed.
Experimental setup
Datasets: PASCAL VOC COCO Object PASCAL Context PASCAL Context59 COCO Stuff PASCAL VOC20 ADE20k Cityscapes
Benchmarks: PASCAL VOC COCO Object PASCAL Context PASCAL Context59 COCO Stuff PASCAL VOC20 ADE20k Cityscapes
Metrics: mIoU Boundary IoU
Baselines: MaskCLIP GEM SCLIP ClearCLIP CLIP-DIY CLIP-DINOiser ProxyCLIP LaVG
Builds on: CLIP-DINOiser MaskCLIP label propagation DINO ProxyCLIP
Closest papers by meaning
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.89)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.88)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.88)
- Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.88)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.87)
- Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.