Research map / Detection and segmentation / Open-set and few-shot detection
DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeCLIP, an unsupervised fine-tuning method that decouples CLIP's last self-attention into content and context features, distilling CLIP self-crop [CLS] alignment into content and VFM (DINOv2) token correlations into context, to improve open-vocabulary dense prediction.
Problem
CLIP's image tokens fail to aggregate information from spatially or semantically related regions, yielding features lacking local discriminability and spatial consistency, which hurts open-vocabulary dense prediction (detection, segmentation).
Approach
Decouple the final self-attention block of CLIP into a "content" branch (aligned to image-crop [CLS] tokens via self-distillation) and a "context" branch (whose attention correlations are distilled from a vision foundation model such as DINOv2), so the two conflicting optimization objectives do not interfere.
Key results
- On OV-COCO novel classes, F-ViT+DeCLIP (ViT-L/14) reaches 46.2 AP50 novel (+1.9) and OV-DQUO+DeCLIP (ViT-L/14) reaches 48.3 (+2.7).
- CAT-Seg+DeCLIP with ViT-L/14 reaches 17.6 mIoU on ADE847 and 97.7 on VOC20 in open-vocabulary semantic segmentation, new SOTA per the paper.
- On VLM-feature segmentation over eight benchmarks, DeCLIP averages 41.9 mIoU versus 40.3 for CLIP-DINOiser.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Decouple self-distillation and VFM distillation features (ablation) | DeCLIP · COCO (Thing) | mAcc: 65.6 -> 75.0 | Helped |
| Decouple self-distillation and VFM distillation features (ablation) | DeCLIP · COCO (Stuff) | mAcc: 41.3 -> 51.8 | Helped |
| Decouple self-distillation and VFM distillation features (ablation) | DeCLIP · Context59 | mIoU: 32.4 -> 35.3 | Helped |
| Add DeCLIP to OV-DQUO | OV-DQUO with CLIP ViT-L/14 · OV-COCO | APNovel50: +2.7 | Helped |
| Add DeCLIP to OV-DQUO | OV-DQUO with CLIP ViT-L/14 · OV-LVIS | mAPr: +2.2 | Helped |
| Use DeCLIP in the LVIS-trained detector | LVIS-trained detector · COCO | AP: 40.5 -> 41.0 | Helped |
| Use DeCLIP in the LVIS-trained detector | LVIS-trained detector · Objects365 | AP: 19.5 -> 20.0 | Helped |
| Replace CAT-Seg backbone with DeCLIP ViT-L/14 | CAT-Seg · ADE847 | mIoU: +1.6 | Helped |
| Use DINOv2 ViT-B/14 as the VFM teacher instead of SAM ViT-B/16 (ablation) | DeCLIP · COCO-Stf | mIoU: 22.0 -> 25.3 | Helped |
| Use DINO ViT-B/16 as the VFM teacher instead of DINOv2 ViT-B/14 (ablation) | DeCLIP · COCO (Thing) | mAcc: 77.2 -> 67.6 | Hurt |
Surprising findings
- Naively combining VFM distillation with self-distillation on the same features lowers region classification accuracy (-3.9 mAcc), motivating decoupling.
Limitations
- Gains on some benchmarks are modest (e.g., +1.5 to +2.3 mAP on OV-LVIS with F-ViT).
- Distillation benefit depends on choice of VFM; SAM and DINO each underperform DINOv2 on some metrics.
Experimental setup
Datasets: COCO LVIS
Benchmarks: OV-COCO OV-LVIS COCO Objects365 ADE20K Pascal Context Pascal VOC Cityscapes COCO-Stuff
Metrics: AP AP50 mIoU mAcc
Baselines: CLIPSelf RegionCLIP F-ViT OV-DQUO CAT-Seg ViLD CLIP-DINOiser ClearCLIP SCLIP FC-CLIP MaskCLIP
Builds on: CLIP CLIPSelf F-ViT CAT-Seg OV-DQUO DINOv2
Closest papers by meaning
- CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction (ICLR 2024 · similarity 0.88)
- ResCLIP: Residual Attention for Training-free Dense Vision-language Inference (CVPR 2025 · similarity 0.88)
- PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts (ICLR 2024 · similarity 0.88)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.87)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.87)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.