Research map / Vision-language and multimodal / CLIP and zero-shot
FG-CLIP: Fine-Grained Visual and Textual Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FG-CLIP, a two-stage CLIP extension trained on 1.6B long-caption pairs and a 12M-image region-grounded dataset (FineHARD) with 10M hard negatives, improving fine-grained, region-level, retrieval and detection performance.
Problem
CLIP struggles with fine-grained understanding because of short 77-token captions, global image-text alignment, and the lack of region-level and hard fine-grained negative supervision.
Approach
Extends CLIP with long-caption global alignment (position embedding interpolation to 248 tokens), region-text contrastive learning via RoIAlign, and hard negative loss built from LLM-generated attribute-swapped region descriptions, packaged in a large dataset (FineHARD).
Key results
- FG-CLIP ViT-L/14 reaches 48.4 hard and 89.7 trivial accuracy on FG-OVD, versus 15.4 and 38.8 for CLIP ViT-L/14.
- On OV-COCO, F-ViT+FG-CLIP ViT-L/14 achieves 41.2 AP50 novel, above F-ViT+CLIPSelf (38.4) and F-ViT+FineCLIP (40.0).
- FG-CLIP ViT-B/16 improves bounding-box classification on COCO to 52.3 top-1 versus 44.2 for CLIP ViT-B/16; LLaVA-v1.5 with FG-CLIP improves RefCOCO val from 76.2 to 81.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add detailed long-caption data and Stage 1 training to CLIP (ablation) | CLIP · DCI I2T | retrieval score: 45.5 -> 58.3 | Helped |
| Add Stage 2 global contrastive learning (ablation) | FG-CLIP Stage1 · DCI I2T | retrieval score: 58.3 -> 62.7 | Helped |
| Add regional contrastive learning (ablation) | FG-CLIP with Stage 2 global contrastive learning · BBox classification | Top-1 accuracy: 46.8% to 53.7% | Helped |
| Add regional contrastive learning (ablation) | FG-CLIP with Stage 2 global contrastive learning · FG-OVD easy | accuracy: 42.9% to 49.5% | Helped |
| Add hard fine-grained negative-sample learning (ablation) | FG-CLIP with global and regional contrastive learning · FG-OVD hard | accuracy: 24.5% to 46.1% | Helped |
| Add hard fine-grained negative-sample learning (ablation) | FG-CLIP with global and regional contrastive learning · FG-OVD medium | accuracy: 47.1% to 66.6% | Helped |
| Add hard fine-grained negative-sample learning (ablation) | FG-CLIP with global and regional contrastive learning · FG-OVD easy | accuracy: 49.5% to 68.7% | Helped |
| Use FG-CLIP instead of FineCLIP with the same FG-CLIP 12M dataset | FineCLIP · COCO bbox classification | Top-1 accuracy: 53.5 -> 56.1 | Helped |
Surprising findings
- Long-CLIP shows a significant drop in bounding-box classification relative to CLIP, which the authors attribute to long texts hurting regional granularity.
- FG-CLIP is marginally behind EVA-CLIP on ImageNet zero-shot classification despite improvements on fine-grained tasks.
Limitations
- Negative sample quality check covers only 3,000 samples (98.9% judged qualified), so label noise is not fully characterized.
- Full 1.6B+12M training was not used for all comparisons; a same-data comparison uses only the 12M set due to time constraints.
- FG-CLIP trails EVA-CLIP on ImageNet zero-shot classification.
Experimental setup
Datasets: LAION-2B GRIT FineHARD ShareGPT4V DCI FG-OVD MSCOCO Flickr30k ImageNet-1K LVIS Open Images GQA
Benchmarks: FG-OVD COCO-val2017 LVIS Open Images OV-COCO ShareGPT4V DCI MSCOCO Flickr30k ImageNet-1K ImageNet-v2 GQA
Metrics: top-1 accuracy AP50 recall@K box classification top-1
Baselines: CLIP EVA-CLIP Long-CLIP FineCLIP RegionCLIP CLIPSelf OV-RCNN Detic VLDet RO-ViT CFM-ViT F-ViT
Builds on: CLIP FineCLIP Long-CLIP RegionCLIP GRIT
Closest papers by meaning
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.92)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.90)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.90)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.90)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.