Research map / Vision-language and multimodal / CLIP and zero-shot
Contrastive Localized Language-Image Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLOC, a CLIP-style pre-training method adding a region-text contrastive loss via a lightweight promptable region extractor, trained on 2B images with region labels produced by a VESL captioning-plus-OWLv2 pseudo-labeling pipeline.
Problem
CLIP aligns whole images with noisy web text, so its embeddings lack the region-level localization needed by MLLMs for referring and grounding tasks, and large-scale region-text training data is scarce.
Approach
Introduce promptable embeddings: the image encoder outputs image tokens from which a lightweight single-layer transformer Prompter extracts region embeddings conditioned on box prompts, trained with a region-text contrastive loss plus a grounding loss on pseudo-labeled region-text data produced by VESL (visually-enriched recaptioning, NER phrase extraction, and OWLv2 detection).
Key results
- With WiT+DFN region labels, CLOC (ViT-B/16) reaches 65.1 average image-task accuracy and 57.3 average region-task score versus 64.3 and no region score for CLIP, and ViT-H/14 reaches 71.7 image and 61.3 region averages.
- On Ferret-Bench, CLOC L/14 with the Prompter as visual sampler replacement scores 56.9 average versus 50.7 for in-house CLIP L/14 (+6.2).
- On LLaVA-1.5 with ViT-L/14, CLOC improves LLaVA-Bench from 59.3 to 64.3 and MME perception from 1451.4 to 1482.0 relative to CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add region-text contrastive training with Prompter (ablation) | CLIP ViT-B/16 · GRIT-t2r | recall@10: 18.4 -> 67.4 | Helped |
| add VESL region annotations (ablation) | CLOC ViT-B/16 · COCO-i2t | recall@1: 53.9 -> 68.8 | Helped |
| use OWLv2 instead of GLIPv2 for region labeling (ablation) | CLOC ViT-B/16 · COCO region recognition | mAcc: 64.9 -> 70.6 | Helped |
| apply text filtering to contrastive negatives (ablation) | CLOC with WiT+DFN image data and WiT region data · COCO-t2i | recall@1: 46.0 -> 46.5 | Helped |
| add grounding loss (ablation) | CLOC with WiT+DFN image data and WiT region data · LVIS region recognition | mAcc: 25.8 -> 27.2 | Helped |
| sample M=2 regions per image instead of M=4 (ablation) | CLOC with WiT+DFN image data and WiT region data · COCO-i2t | recall@1: 66.6 vs. 66.1 | No clear effect |
| replace Prompter with RoI-Align (ablation) | CLOC in Ferret · Ferret-Bench | average: 47.7 -> 45.5 | Hurt |
| replace Ferret visual sampler with Prompter (ablation) | Ferret with CLOC ViT-B/16 · Ferret-Bench | average: 49.8 -> 51.5 | Helped |
| scale region labels from 300M to 2B images | CLOC in Ferret · Ferret-Bench | average: 49.1 -> 49.8 | Helped |
| replace CLIP encoder with CLOC encoder | Ferret ViT-B/16 · RefCOCO val | accuracy: 80.7 -> 84.2 | Helped |
| replace CLIP encoder with CLOC encoder | LLaVA-1.5 with ViT-L/14 · LLaVA-W | score: 59.3 -> 64.3 | Helped |
| replace CLIP encoder with CLOC encoder | Open-LLaVA-NeXT with ViT-L/14 · MMEp | score: 1486.1 -> 1451.1 | Hurt |
Surprising findings
- Replacing the Prompter with RoI-Align causes a large drop in region retrieval (GRIT t2r falls from 67.2 to 4.4 without the Prompter), and RoI-Align can slightly degrade Ferret-Bench performance.
- Fine-tuning an existing CLIP checkpoint with the CLOC loss performs worse than training from scratch (64.1% vs 70.1% COCO region mAcc), even with more total steps.
Limitations
- Annotations and checkpoints are only promised for future release; labeling at billion scale needs hundreds of GPUs for days.
- Minimal tuning of hyper-parameters, architecture, and dataset cleaning, so performance could be improved.
- Only box and text prompts are explored; other prompt types are left for future work.
Experimental setup
Datasets: WiT-300M DFN-5B GRIT COCO LVIS RefCOCO Flickr30k Entities ImageNet
Benchmarks: COCO retrieval ImageNet classification GRIT region retrieval COCO region recognition LVIS region recognition Ferret-Bench RefCOCO RefCOCO+ RefCOCOg Flickr30k Entities LLaVA-Bench TextVQA
Metrics: recall@1 recall@10 accuracy mAcc accuracy at 0.5 IoU
Baselines: OpenAI-CLIP CLIP (in-house) RoI-Align region variant GLIPv2 pseudo-labeling variant Ferret Shikra LLaVA-1.5 Open-LLaVA-NeXT
Builds on: CLIP SAM OWLv2 VeCap Ferret
Closest papers by meaning
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.89)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.89)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.89)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.89)
- Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning (AAAI 2024 · similarity 0.88)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.