Research map / Detection and segmentation / Open-set and few-shot detection
Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CASS, a training-free open-vocabulary semantic segmentation method that distills low-rank spectral graph structure from DINO's attention into CLIP's attention via spectral graph matching, and refines text embeddings and patch-text similarity with CLIP's object presence prior.
Problem
Training-free open-vocabulary semantic segmentation with CLIP lacks object-level context, so components of the same object are not grouped into one user-defined class.
Approach
Treat the VFM (DINO) attention heads as graphs, match heads to CLIP heads by Wasserstein distance between eigenvalue spectra, distill a low-rank, dynamically eigen-scaled VFM graph into CLIP's attention, and adjust text embeddings and patch-text similarity using CLIP's image-level object presence prior.
Key results
- CASS achieves 44.4 average mIoU across eight datasets with ViT-B/16, a 3.0 point gain over the second-best method.
- CASS reaches 65.8 mIoU on V21 and 87.8 on V20 with CLIP ViT-B/16.
- CASS attains 68.9 average pixel accuracy, versus 66.7 for ProxyCLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add spectral VFM distillation with graph matching, low-rank approximation, and dynamic eigenscaling (ablation) | CASS · PASCAL VOC 2012 with background (V21) | mIoU: 58.9 -> 64.8 | Helped |
| Add spectral VFM distillation with graph matching, low-rank approximation, and dynamic eigenscaling (ablation) | CASS · PASCAL Context without background (PC59) | mIoU: 35.3 -> 38.6 | Helped |
| Add spectral VFM distillation with graph matching, low-rank approximation, and dynamic eigenscaling (ablation) | CASS · COCO stuff (C-Stf) | mIoU: 23.4 -> 25.9 | Helped |
| Add object-presence-prior methods (OPS and OTA) (ablation) | CASS with spectral VFM distillation · PASCAL VOC 2012 with background (V21) | mIoU: 64.8 -> 65.8 | Helped |
| Add object-presence-prior methods (OPS and OTA) (ablation) | CASS with spectral VFM distillation · PASCAL Context without background (PC59) | mIoU: 38.6 -> 40.2 | Helped |
| Add object-presence-prior methods (OPS and OTA) (ablation) | CASS with spectral VFM distillation · COCO stuff (C-Stf) | mIoU: 25.9 -> 26.7 | Helped |
| Add object-presence-prior methods (OPS and OTA) (ablation) | Vanilla CLIP segmentation · PASCAL VOC 2012 with background (V21) | mIoU: 58.9 -> 59.2 | Helped |
| Add object-presence-prior methods (OPS and OTA) (ablation) | Vanilla CLIP segmentation · PASCAL Context without background (PC59) | mIoU: 35.3 -> 36.5 | Helped |
| Add object-presence-prior methods (OPS and OTA) (ablation) | Vanilla CLIP segmentation · COCO stuff (C-Stf) | mIoU: 23.4 -> 24.0 | Helped |
| Use Wasserstein distance instead of the other evaluated distance metrics for graph matching (ablation) | CASS · PASCAL VOC 2012 with background (V21) | mIoU | Helped |
| Use CASS instead of the strongest baseline reported | CLIP ViT-B/16 · PASCAL VOC 2012 with background (V21) | mIoU: 62.1 -> 65.8 | Helped |
| Use CASS instead of the strongest baseline reported | CLIP ViT-B/16 · COCO with object classes (C-Obj) | mIoU: 43.6 -> 37.8 | Hurt |
Limitations
- Relies on frozen DINO and CLIP backbones, so performance depends on the quality of the VFM attention graph.
- Sliding-window inference and multiple hyperparameters (k, alpha, gamma, top-n) add tuning choices not fully explored in the main text.
Experimental setup
Datasets: PASCAL VOC 2012 PASCAL Context COCO ADE20K Cityscapes
Benchmarks: PASCAL VOC 2012 PASCAL Context COCO-Object COCO-Stuff ADE20K Cityscapes VOC20 PC59
Metrics: mIoU pAcc
Baselines: NACLIP ProxyCLIP SCLIP LaVG ClearCLIP CLIPtrase CLIP-DINOiser MaskCLIP
Builds on: CLIP DINO NACLIP ProxyCLIP SCLIP
Closest papers by meaning
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.91)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.90)
- Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation (NeurIPS 2025 · similarity 0.89)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.89)
- Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation (NeurIPS 2024 · similarity 0.89)
- Open-Vocabulary Segmentation with Semantic-Assisted Calibration (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.