Research map / Vision-language and multimodal / CLIP and zero-shot
Refining CLIP's Spatial Awareness: A Visual-Centric Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spatial Correlation Distillation (SCD) and a self-supervised Refiner to preserve and enhance CLIP ViTs' spatial awareness while fine-tuning with region-language alignment, improving open-vocabulary detection and segmentation.
Problem
Region-language alignment (RLA) fine-tuning of CLIP ViTs degrades visual-centric spatial awareness of dense features, which hurts open-vocabulary dense prediction tasks.
Approach
Distill intra-image spatial correlation matrices from the original CLIP ViT into the RLA-tuned student (SCD), and train a lightweight Refiner (initialized from the last CLIP blocks) with a global-to-local InfoNCE objective to produce cleaner dense targets for the distillation.
Key results
- R-SC-CLIPSelf with ViT-L/14 reaches 48.1 AP50 novel on OV-COCO, versus 44.3 for CLIPSelf.
- Cat-Seg with R-SC-V on ViT-L/14 improves ADE-150 mIoU from 37.9 to 38.4 and ADE-847 from 16.0 to 16.6.
- On stuff masks zero-shot Top1 with ViT-B/16 and RPN proposals, R-SC-CLIPSelf improves over CLIPSelf from 36.8 to 52.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add refined spatial correlation distillation to CLIPSelf | CLIPSelf, ViT-L/14 · OV-COCO | AP50 novel: 44.3 -> 48.1 | Helped |
| Add refined spatial correlation distillation to CLIPSelf | CLIPSelf, ViT-L/14 · OV-LVIS | rare-class mAP: 34.9 -> 37.2 | Helped |
| Add R-SC-CLIPSelf to vanilla Cat-Seg | Cat-Seg†, CLIP ViT-B/16 · ADE-150 | mIoU: 27.2 -> 29.9 | Helped |
| Use SCD without the Refiner (ablation) | CLIPSelf, ViT-B/16 · OV-COCO | AP50 novel: 37.6 -> 39.1 | Helped |
| Train Refiner and student end-to-end rather than in two stages (ablation) | R-SC-CLIPSelf, ViT-B/16 · OV-COCO | AP50 novel: 40.9 -> 40.0 | Hurt |
| Reverse global-to-local Refiner training to local-to-global (ablation) | R-SC-CLIPSelf, ViT-B/16 · OV-COCO | AP50 novel: 39.1 -> 38.0 | Hurt |
| Initialize Refiner randomly instead of inheriting CLIP weights (ablation) | R-SC-CLIPSelf, ViT-B/16 · OV-COCO | AP50 novel: 40.9 -> 39.4 | Hurt |
| Replace SCD cross-entropy with Frobenius norm loss (ablation) | CLIPSelf, ViT-B/16 · OV-COCO | AP50 novel: 37.6 -> 36.8 | Hurt |
| Use inter-instance correlations instead of within-scene correlations (ablation) | CLIPSelf, ViT-B/16 · OV-COCO | AP50 novel: 37.6 -> 37.2 | Hurt |
| Use dense-level contrastive visual constraint (ablation) | CLIPSelf, ViT-B/16 · OV-COCO | AP50 novel: 37.6 -> 27.4 | Hurt |
| Use masked image modeling visual constraint (ablation) | CLIPSelf, ViT-B/16 · OV-COCO | AP50 novel: 37.6 -> 37.5 | No clear effect |
| Distill correlations over the whole image instead of local regions (ablation) | CLIPSelf, ViT-B/16 · OV-COCO | AP50 novel: 40.9 -> 39.7 | Hurt |
Surprising findings
- RLA fine-tuning (RegionCLIP, CLIPSelf) degrades visual-centric dense feature quality, and typical visual-centric constraints (MIM, dense contrastive) fail to help under RLA fine-tuning.
- Fine-tuning the text encoder in the updated Cat-Seg lowers CLIPSelf performance, which the authors attribute to reduced CLIPSelf advantage and weaker spatial awareness.
Limitations
- CLIP's native spatial awareness remains limited, capping the benefit of SC-RLA without the Refiner.
- Aggregating many context images for training-free refinement is computationally expensive and impractical at inference, motivating the Refiner.
- Gains on OV-COCO/LVIS are moderate (typically 2-4 points) and the method was mostly evaluated with short fine-tuning on COCO.
Experimental setup
Datasets: COCO train2017 ADE20K CC3M COCO Panoptic Cityscapes COCO-Stuff Pascal Context
Benchmarks: OV-COCO OV-LVIS ADE-150 ADE-847 PASCAL Context Cityscapes unsupervised segmentation (CAUSE) COCO-Stuff
Metrics: AP50 novel mAP_r mIoU mACC Top1 zero-shot dense accuracy CR coupling ratio
Baselines: CLIPSelf RegionCLIP RegionText EVA-CLIP Cat-Seg F-VLM BARON-KD CORA+ DITO CFM-ViT RO-ViT SILC
Builds on: CLIPSelf RegionCLIP Cat-Seg F-ViT MaskCLIP
Closest papers by meaning
- SpatialCLIP: Learning 3D-aware Image Representations from Spatially Discriminative Language (CVPR 2025 · similarity 0.91)
- Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling (NeurIPS 2026 · similarity 0.90)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
- ResCLIP: Residual Attention for Training-free Dense Vision-language Inference (CVPR 2025 · similarity 0.89)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.88)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.