Research map / Vision-language and multimodal / CLIP and zero-shot
FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends CLIP to 248-token text inputs by stretching positional embeddings and adds adaptive token aggregation plus token-level triplet-loss alignment, fine-tuned on ShareGPT4V, evaluated on long-caption retrieval (Urban1k, DOCCI) and SDXL long-text generation.
Problem
CLIP text encoders are limited to 77 tokens and align only global features, so they handle long, detail-rich captions poorly and miss fine-grained image-text correspondences.
Approach
Stretch CLIP's positional embeddings to 248 tokens, learn soft token aggregation (ATRM) for both image and text tokens, and apply a bidirectional max-sim token-to-token late-interaction triplet loss (CLIM) that also includes global CLS/EOS tokens.
Key results
- On Urban1k with B/16, FineLIP reaches 0.907 I2T R@1 vs 0.859 for the coarse-aligned Baseline and 0.890 for LAPS.
- With L/14 on Urban1k, FineLIP* (weighted global+local inference) reaches 0.945 I2T R@1 vs 0.892 for Baseline.
- In SDXL long-text generation with L/14, FineLIP achieves FID 27.261 on Urban1k vs 29.535 for Baseline, though LAPS obtains a better 26.743.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FineLIP* instead of LAPS | CLIP ViT-L/14 retrieval · Urban1k | I2T Recall@1: 0.921 -> 0.945 | Helped |
| Use FineLIP* instead of LAPS | CLIP ViT-L/14 retrieval · Urban1k | T2I Recall@1: 0.916 -> 0.939 | Helped |
| Use FineLIP* instead of LAPS | CLIP ViT-L/14 retrieval · DOCCI | I2T Recall@1: 0.818 -> 0.837 | Helped |
| Use FineLIP* instead of LAPS | CLIP ViT-L/14 retrieval · DOCCI | T2I Recall@1: 0.818 -> 0.844 | Helped |
| Replace Baseline with FineLIP | SDXL text-to-image generation · Urban1k | FID: 29.535 -> 27.261 | Hurt |
| Replace Baseline with FineLIP | SDXL text-to-image generation · DOCCI | FID: 16.884 -> 15.410 | Helped |
| Add ATRM to CLIM, refining image tokens (ablation) | FineLIP retrieval · Urban1k | T2I Recall@1: 0.782 -> 0.890 | Helped |
| Add ATRM to CLIM, refining text tokens (ablation) | FineLIP retrieval · Urban1k | T2I Recall@1: 0.782 -> 0.827 | Helped |
| Refine both image and text tokens with ATRM (ablation) | FineLIP retrieval · Urban1k | I2T Recall@1: 0.854 -> 0.907 | Helped |
| Set ATRM aggregation ratio to 0.1 instead of 0.2 (ablation) | FineLIP retrieval · DOCCI | I2T Recall@1: 0.771 -> 0.776 | Helped |
| Set ATRM aggregation ratio to 0.4 instead of 0.2 (ablation) | FineLIP retrieval · DOCCI | T2I Recall@1: 0.795 -> 0.785 | Hurt |
| Retain global [CLS] and [EOS] tokens in alignment (ablation) | FineLIP retrieval · DOCCI | T2I Recall@1: 0.773 -> 0.795 | Helped |
Surprising findings
- Long-CLIP performs worse than the simple stretched-positional-embedding Baseline on long-caption retrieval.
- Refining only text tokens yields smaller gains than refining only image tokens, attributed to text tokens' higher information density.
Limitations
- Performance on short-caption retrieval and classification is less satisfactory.
- Relies on ShareGPT4V long captions for training.
- Code is promised but the paper gives no release statement beyond the future availability note.
Experimental setup
Datasets: ShareGPT4V
Benchmarks: Urban1k DOCCI
Metrics: Recall@1 Recall@5 Recall@10 FID
Baselines: Baseline (CLIP with stretched positional embeddings, coarse contrastive) SPARC LAPS Long-CLIP TULIP
Builds on: CLIP Long-CLIP FILIP SPARC LAPS
Closest papers by meaning
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.92)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.91)
- TULIP: Token-length Upgraded CLIP (ICLR 2025 · similarity 0.91)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.91)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.90)
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.