Research map / Vision-language and multimodal / CLIP and zero-shot
FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FineCLIP trains CLIP end-to-end with global contrastive loss, a real-time self-distillation loss from global crop embeddings to pooled region features, and a regional contrastive loss on LVLM-generated region-text pairs to improve fine-grained and dense prediction performance.
Problem
CLIP's global image-text alignment yields weak fine-grained region representations, hurting dense prediction tasks such as box classification, open-vocabulary detection and segmentation, while prior fixes either use limited template labels or a frozen teacher that caps performance.
Approach
Combine global contrastive learning with a real-time self-distillation scheme in which the trainable model distills its own current global crop embeddings into pooled region features, plus regional contrastive learning on region descriptions generated by LVLMs (BLIP-2) instead of template labels.
Key results
- On COCO box classification with ViT-B/16, FineCLIP reaches 48.4 Top1 vs 42.3 for CLIP and 43.7 for CLIPSelf in the ablation setting.
- With CC2.5M training, FineCLIP gains +10.4 mAP over CLIP and +8.0 over CLIPSelf on zero-shot box classification (50.7 Top1 at 2.5M samples).
- On OV-COCO with F-ViT ViT-L/14, FineCLIP achieves 40.0 APnovel50 vs 38.4 for CLIPSelf; CatSeg ViT-L/14 with FineCLIP reaches 36.1 ADE-150 mIoU vs 34.9 for CLIPSelf.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add regional contrastive loss to global contrastive learning (ablation) | FineCLIP · COCO val2017 box classification | Top1 mean accuracy: 42.3 -> 47.8 | Helped |
| Add regional contrastive loss to global contrastive learning (ablation) | FineCLIP · COCO val2017 box classification | Top5 mean accuracy: 66.6 -> 74.1 | Helped |
| Add self-distillation loss to global contrastive learning (ablation) | FineCLIP · COCO val2017 box classification | Top1 mean accuracy: 42.3 -> 43.7 | Helped |
| Add self-distillation loss to global contrastive learning (ablation) | FineCLIP · COCO val2017 image-to-text retrieval | R@1: 62.4 -> 60.0 | Hurt |
| Remove global contrastive supervision, leaving self-distillation only (ablation) | FineCLIP · COCO val2017 box classification | Top1 mean accuracy: 42.3 -> 0.0 | Hurt |
| Use LVLM-generated region descriptions instead of rule-based annotations (ablation) | FineCLIP · COCO val2017 box classification | Top1 mean accuracy: 43.1 -> 48.4 | Helped |
| Use YOLOv9 proposals instead of manual region proposals (ablation) | FineCLIP · COCO val2017 box classification | Top1 mean accuracy: 48.4 -> 49.6 | Helped |
| Use FastSAM proposals instead of manual region proposals (ablation) | FineCLIP · COCO val2017 box classification | Top1 mean accuracy: 48.4 -> 47.1 | Hurt |
| Increase input image size from 224 to 640 (ablation) | FineCLIP · COCO val2017 box classification | Top1 mean accuracy: 48.4 -> 54.3 | Helped |
| Increase input image size from 224 to 640 (ablation) | FineCLIP · COCO val2017 retrieval | R@1 image-to-text: 62.2 -> 60.7 | Mixed |
| Replace ViT-B/16 with ViT-L/14 (ablation) | FineCLIP · COCO val2017 box classification | Top1 mean accuracy: 48.4 -> 52.6 | Helped |
| Replace ViT-B/16 with ViT-L/14 (ablation) | FineCLIP · COCO val2017 image-to-text retrieval | R@1: 62.2 -> 66.0 | Helped |
Surprising findings
- Self-distillation loss alone causes training collapse (near-zero accuracy) without the global contrastive loss.
- Automated region proposals (e.g., YOLOv9, RPN) perform comparably to manual COCO annotations, and more proposals do not necessarily help.
Limitations
- Requires strict region-level data, and region proposal quality strongly affects performance.
- Cannot afford training from scratch on billion-scale datasets, so full potential is not shown.
- Higher GPU memory usage than competing methods, and the paper uses a weaker LVLM (BLIP-2) for annotation due to compute constraints.
Experimental setup
Datasets: COCO train2017 CC3M CC2.5M COCO Stuff OV-COCO OV-LVIS ImageNet
Benchmarks: COCO box classification COCO image-text retrieval OV-COCO OV-LVIS COCO transfer Object365 ADE-150 ADE-847 PASCAL Context Flickr30k MSCOCO
Metrics: Top1 accuracy Top5 accuracy R@1 AP50 mAP mIoU mAcc
Baselines: CLIP RegionCLIP CLIPSelf SPARC PACL GLoRIA MGCA FILIP OVSeg SAN CatSeg F-VLM
Builds on: CLIP CLIPSelf RegionCLIP F-ViT CatSeg
Closest papers by meaning
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.91)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.90)
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.90)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.