Research map / Vision-language and multimodal / CLIP and zero-shot

Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models

ICML 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Open-vocabulary alignment from VLMs · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes weighted visual-text cross alignment (WCA), which randomly crops local patches of an image, cross-aligns them with LLM-generated class descriptions in CLIP space, and aggregates the similarity matrix with cosine-based weights for training-free zero-shot classification.

Problem

Zero-shot VLM classification with LLM-generated fine-grained descriptions aligns descriptions with the whole image, but finer descriptions often match local image regions better, lowering global similarity.

Approach

Replace global image-description averaging with a weighted cross-alignment over random image crops and LLM descriptions, where weights are the softmax of cosine similarities of each crop to the whole image and of each description to the base class prompt, so that informative patches and relevant descriptions dominate the score; no training or extra models are needed.

Key results

• WCA improves ImageNet top-1 accuracy with CLIP ViT-B/32 from 64.37 (CuPL) to 66.84, and CUB by +4.17 over the best baseline with B/32.
• On ImageNet domain shift benchmarks with CLIP L/14, WCA achieves the highest average accuracy of 64.03 across target sets, versus 62.58 for DiffTPT, without any tuning.
• Ablation on ImageNet shows the full method (crops, LLM descriptions, both weights) gives +2.60 over the CLIP baseline with B/32, compared with +1.12 for descriptions alone.

Measured effects reported in the paper

Surprising findings

• Naive max or mean aggregation over random crops can fall below the CLIP baseline on ImageNet, showing that weighting is necessary.
• Red Circle and Blur visual prompts slightly reduce accuracy relative to the CLIP baseline, while random cropping improves it.

Limitations

• Performance is limited on holistic tasks such as EuroSAT that require whole-image understanding.
• Small objects among larger ones can be underweighted by patch-to-image similarity weights.
• Text weights rely on cosine similarity to a base prompt, which may be sub-optimal.

Experimental setup

Datasets: ImageNet CUB Oxford Pets DTD Food101 Place365 ImageNet-V2 ImageNet-R ImageNet-Sketch ImageNet-A LLM-generated class descriptions (CuPL files)

Benchmarks: ImageNet CUB Oxford Pets DTD Food101 Place365 ImageNet-V2 ImageNet-R ImageNet-Sketch ImageNet-A Flowers102 EuroSAT

Metrics: top-1 accuracy

Baselines: CLIP CLIP-E CLIP-D CuPL Waffle CoOp CoCoOp UPT ProGrad KgCoOp MaPLe TPT

Builds on: CLIP Visual-Text Alignment via LLMs (CuPL/CLIP-D) test-time prompt tuning

Closest papers by meaning

• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.90)
• Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.90)
• GOAL: Global-local Object Alignment Learning (CVPR 2025 · similarity 0.89)
• Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.89)
• Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models (ICML 2025 · similarity 0.89)
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.89)