Research map / Detection and segmentation / Open-set and few-shot detection
CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIPSelf, a region-free self-distillation method that aligns ViT dense region features with the CLIP image-level embeddings of image crops, improving open-vocabulary detection and segmentation.
Problem
CLIP ViTs transfer poorly from global image representations to local region representations, so their dense features are weak for open-vocabulary dense prediction, and region-text pair supervision is costly or noisy.
Approach
A teacher-student self-distillation where the frozen original CLIP ViT encodes image-patch crops and the fine-tuned student's RoI/mask-pooled dense features are aligned to those crop embeddings via cosine loss, requiring no region-text pairs.
Key results
- F-ViT+CLIPSelf with ViT-L/14 reaches 44.3 AP50 novel on OV-COCO versus 24.7 for F-ViT without CLIPSelf, and 34.9 mAP_r on OV-LVIS versus 24.2.
- CLIPSelf on ViT-B/16 (EVA-CLIP) raises region box Top1 classification accuracy from 18.2% to 72.1%.
- Cat-Seg+CLIPSelf with ViT-L/14 improves ADE-150 mIoU from 31.5 to 34.5 and PASCAL Context mIoU from 62.0 to 62.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CLIPSelf to F-ViT | EVA-CLIP ViT-L/14 detector · OV-COCO | AP50 novel: 24.7 vs 44.3 | Helped |
| Add CLIPSelf to F-ViT | EVA-CLIP ViT-L/14 detector · OV-LVIS | mAP rare: 24.2 vs 34.9 | Helped |
| Add CLIPSelf to Cat-Seg | OpenAI ViT-L/14 · ADE-150 | mIoU: 31.5 -> 34.5 | Helped |
| Add CLIPSelf to ODISE and fuse CLIP and mask-generator scores | OpenAI ViT-L/14 · ADE20K | mIoU: 28.5 -> 30.1 | Helped |
| Use patch-split maximum M=6 instead of M=4, M=8, or M=10 (ablation) | CLIPSelf on EVA-CLIP ViT-B/16 · COCO val2017 region boxes | Top1 mAcc: 71.3, 71.6, and 70.0 -> 72.1 | Helped |
| Train more attention layers (ablation) | CLIPSelf on EVA-CLIP ViT-B/16 · COCO val2017 region boxes | Top1 mAcc: 45.0, 59.4, 68.7 -> 72.1 | Helped |
| Increase student input size (ablation) | CLIPSelf on EVA-CLIP ViT-B/16 · COCO val2017 region boxes | Top1 mAcc: 46.5 -> 67.1 -> 72.1 | Helped |
| Replace region-wise self-distillation with image-level supervision (ablation) | EVA-CLIP ViT-B/16 at student input size 1024 · COCO val2017 region boxes | Top1 mAcc: 72.1 -> 52.3 | Hurt |
| Use region proposals instead of image patches (ablation) | CLIPSelf on EVA-CLIP ViT-B/16 · COCO val2017 boxes and stuff masks | Top1 mAcc: Boxes: 72.1 -> 74.0; stuff masks: 46.8 -> 36.8 | Mixed |
| Replace global attention with 4×4 window attention (ablation) | EVA-CLIP ViT-B/16 without CLIPSelf · COCO val2017 boxes | Top1 mAcc: 18.2 -> 34.7 | Helped |
| Add CLIPSelf to the 4×4 WindowAttn model (ablation) | EVA-CLIP ViT-B/16 · COCO val2017 boxes | Top1 mAcc: 34.7 -> 73.3 | Helped |
| Use masked attention to extract dense features (ablation) | OpenAI ViT-B/16 · COCO region recognition | Top1 mAcc: Boxes: 29.3 -> 28.2; stuff masks: 25.9 -> 32.2 | Mixed |
Surprising findings
- Image-level supervision on COCO Caption at the same 1024 input size only raises Top1 region accuracy to 52.1%, far below self-distillation, so the gain is not just from higher resolution.
- Unfreezing the ViT backbone in the detector improves base-category AP50 but decreases novel-category AP50.
Limitations
- Performance is bounded by the alignment of the original CLIP model being distilled.
- Not validated on Swin-based detection foundation models (GLIP, Grounding DINO) due to lack of public CLIP-like Swin models.
- Region-proposal variant helps foreground but weakens stuff recognition, so it is only used for detection.
Experimental setup
Datasets: COCO train2017 LVIS v1.0 train COCO Caption CC3M COCO Stuff COCO Panoptic ADE20K PASCAL Context OV-COCO OV-LVIS
Benchmarks: OV-COCO OV-LVIS PASCAL VOC COCO Objects365 ADE-150 ADE-847 PASCAL Context COCO Panoptic ADE20K
Metrics: AP50 novel mAP rare mIoU mAcc PQ mAP Top1 mean accuracy Top5 mean accuracy
Baselines: ViLD Detic F-VLM OV-DETR BARON-KD CORA CORA+ PromptOVD RO-ViT CFM-ViT SAN Cat-Seg
Builds on: CLIP F-VLM Cat-Seg ODISE RegionCLIP
Closest papers by meaning
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.89)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.89)
- DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception (CVPR 2025 · similarity 0.88)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.88)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.87)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.