Research map / Vision-language and multimodal / CLIP and zero-shot

GOAL: Global-local Object Alignment Learning

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Fine-grained improvements to CLIP · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Fine-tunes CLIP on long detailed captions by pairing SAM-segmented image regions with caption sentences (LISM) and aligning patch/sequence tokens to local CLS embeddings (TSL), evaluated on DOCCI, DCI and Urban1k retrieval.

Problem

CLIP is trained on short captions and struggles to align images with lengthy, detailed text descriptions, especially at local (object/sentence) level.

Approach

Generate pseudo local image-sentence pairs automatically via SAM segmentation and CLIP CLS similarity (LISM), then fine-tune CLIP with global and local contrastive losses plus a token-similarity loss that pulls pooled patch/sequence tokens of matched regions toward local CLS embeddings (TSL).

Key results

• On DOCCI with ViT-L/14, GOAL reaches 84.37% text-to-image R@1 vs 74.00% for global fine-tuning.
• Zero-shot on Urban1k with ViT-L/14, GOAL (DCI fine-tuning) reaches 89.80% T2I R@1 vs 82.60% for Long-CLIP (I2T R@1 not directly compared here).
• On DCI with ViT-L/14, GOAL achieves 76.89% R@1 vs 65.73% for global fine-tuning.

Measured effects reported in the paper

Surprising findings

• Local-only fine-tuning performs worse than global fine-tuning on the original retrieval test sets.

Limitations

• Evaluated only on retrieval benchmarks built from DOCCI, DCI and Urban1k; the global-local test set is pseudo-labeled by the same LISM pipeline.
• Local pairs depend on SAM segmentation and CLIP matching quality; segments under 1% of image area are discarded.

Experimental setup

Datasets: DOCCI DCI Urban1k COCO Flickr30k ShareGPT4V

Benchmarks: DOCCI DCI Urban1k

Metrics: Recall@K mAP@10

Baselines: CLIP Long-CLIP Global fine-tuning Local fine-tuning w/o TSL

Builds on: CLIP Long-CLIP SAM

Closest papers by meaning

• Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding (NeurIPS 2026 · similarity 0.91)
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
• FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.90)
• Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.89)
• Retaining Knowledge and Enhancing Long-Text Representations in CLIP through Dual-Teacher Distillation (CVPR 2025 · similarity 0.89)
• FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.89)