Research map / Vision-language and multimodal / CLIP and zero-shot
FLAIR: VLM with Fine-grained Language-informed Image Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a CLIP-style ViT-B/16 vision-language model on 30M re-captioned image-text pairs using text-conditioned attention pooling over local image tokens and diverse sub-caption sampling, yielding fine-grained image-text retrieval and zero-shot segmentation gains.
Problem
CLIP matches images and texts only at a global level, so it captures detailed local visual features poorly, limiting fine-grained retrieval and localization.
Approach
Use diverse sub-captions from long synthetic descriptions as queries in a text-conditioned attention pooling over local image tokens, producing a per-caption image embedding, trained with a text-conditioned sigmoid loss using negatives chosen so that both image and text must be matched, plus a multi-positive sigmoid loss for global alignment.
Key results
- FLAIR trained on 30M pairs reaches 81.1% T2I R@1 on Flickr30k vs 75.6% for SigLIP-10B.
- On zero-shot semantic segmentation, FLAIR-30M averages 25.8 mIoU versus 14.4 points average gain over billion-scale CLIP models (14.4% average improvement).
- On long-caption retrieval, FLAIR improves T2I R@1 by 10.4%, 5.7%, and 8.2% over prior SOTA on three of the datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace global loss only with text-conditioned loss (ablation) | FLAIR · COCO retrieval | T2I R@1: 28.3 -> 32.0 | Helped |
| Add multiple captions to text conditioning (ablation) | FLAIR · COCO retrieval | T2I R@1: 32.0 -> 34.8 | Helped |
| Add diverse caption sampling to text conditioning and multiple captions (ablation) | FLAIR · VOC20 | mIoU: 34.1 -> 46.5 | Helped |
| Add global loss to text conditioning, multiple captions, and diverse sampling (ablation) | FLAIR · VOC20 | mIoU: 46.5 -> 59.7 | Helped |
| Use diverse sampling rather than a single sentence (ablation) | FLAIR · long retrieval, Urban-1k | T2I R@1: +20% | Helped |
Surprising findings
- Standard CLIP and SigLIP trained on YFCC15M-recap can match or surpass counterparts trained on billions of samples, suggesting long synthetic captions are highly beneficial.
- Zero-shot classification still lags billion-scale CLIP models by around 20% in retrieval-comparable settings, indicating image scale remains key for global classification.
Limitations
- Zero-shot classification lags behind models trained on much larger datasets.
- Trained only on 30M image-text pairs with ViT-B/16; scaling to larger datasets and models is left to future work.
Experimental setup
Datasets: CC3M-recap CC12M-recap YFCC15M-recap Merged-30M MSCOCO Flickr30k DOCCI IIW DCI ShareGPT-4V Urban-1k VOC20
Benchmarks: MSCOCO Flickr30k DOCCI-FG IIW-FG DCI SV-1k SV-10k Urban-1k VOC20 Cityscapes Context59 ADE20K
Metrics: R@1 R@5 mIoU top-1 accuracy
Baselines: CLIP SigLIP DreamLIP LaCLIP Llip Long-CLIP LoTLIP MetaCLIP OpenCLIP
Builds on: CLIP SigLIP DreamLIP OpenCLIP
Closest papers by meaning
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.89)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.89)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.88)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.88)
- LensVLM: Selective Context Expansion for Compressed Visual Representation of Text (NeurIPS 2026 · similarity 0.86)
- Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.