Research map / Vision-language and multimodal / Cross-modal learning
Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CODER, a training-free cross-modal neighbor representation for CLIP images built from image-to-text cosine similarities over LLM-generated texts, used in a two-stage zero-shot classifier and a Tip-Adapter variant for few-shot classification.
Problem
CLIP image encoder features are used directly for intra-modal tasks such as few-shot classification, which misaligns with CLIP's image-text contrastive pre-training and can yield poor image-image distance relations.
Approach
Represent each image by its cosine similarities to a dense set of LLM-generated texts (CODER), treating texts as neighbors in CLIP space; an Auto Text Generator (ChatGPT prompts plus WordNet) produces class-name, attribute, analogous-class, synonym and one-to-one texts without training or image data.
Key results
- Zero-shot ViT-B/32 on ImageNet improves from 59.07 (CLIP) and 62.96 (VCD) to 66.86 with CODER with rerank.
- EuroSAT with ViT-L/14@336px improves by 10.50 points over VCD (54.93 to 69.96 with rerank... gain over VCD reported as 10.50).
- CODER-Adapter surpasses Tip-Adapter and TIP-X on most of 11 datasets in the training-free few-shot regime, but underperforms on EuroSAT (only 95 generated texts).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CLIP image features with CODER and rerank with one-to-one specific CODER | CLIP zero-shot classification (ViT-B/32) · ImageNet | accuracy: 62.96 -> 66.86 | Helped |
| rerank preliminary predictions using one-to-one specific CODER | CODER zero-shot classification (ViT-B/32) · Places365; Food101; Caltech101; Oxford Pets; ImageNetV2 | accuracy: 40.37 -> 40.27; 84.72 -> 85.50; 91.24 -> 91.42; 88.47 -> 89.26; 56.75 -> 56.42 | Mixed |
| add attribute-based texts to class name-based texts (ablation) | CODER zero-shot classification (CLIP ViT-B/32) · Caltech101; Oxford Pets; Describable Textures; EuroSAT | accuracy: 79.90 -> 89.80; 81.63 -> 85.91; 41.28 -> 45.27; 44.89 -> 46.57 | Helped |
| add analogous class-based texts (ablation) | CODER zero-shot classification (CLIP ViT-B/32) · Caltech101; Oxford Pets; Describable Textures; EuroSAT | accuracy: 89.80 -> 91.01; 85.91 -> 88.55; 45.27 -> 48.73; 46.57 -> 52.77 | Helped |
| add analogous class-based texts (ablation) | CODER few-shot classification (CLIP ResNet-50, 16-shot) · Caltech101; Describable Textures; EuroSAT | accuracy: 90.47 -> 90.99; 63.06 -> 64.18; 65.36 -> 67.11 | Helped |
| add analogous class-based texts (ablation) | CODER few-shot classification (CLIP ResNet-50, 16-shot) · Oxford Pets | accuracy: 90.3 -> 90.26 | Hurt |
| increase diversity and number of neighbor texts (ablation) | CODER · zero-shot and few-shot image classification | accuracy | Helped |
| use CODER instead of original CLIP image features for support-set similarity | CODER-Adapter · 11 few-shot image classification datasets | accuracy | Helped |
| use CODER-Adapter with a small text set | CODER-Adapter · EuroSAT few-shot classification | accuracy | Hurt |
Surprising findings
- On EuroSAT, the small number of generated texts (95 for 10 classes) limits CODER, showing dense text sampling is necessary.
Limitations
- Dependence on the quality and density of LLM-generated texts; small class sets yield few texts and weaker gains.
- Two-stage rerank requires generating one-to-one texts for top-5 class pairs, adding inference cost.
Experimental setup
Datasets: ImageNet CUB200 EuroSAT Describable Textures Places365 Food101 Caltech101 Oxford Pets ImageNetV2 SUN397 Cars Flowers102
Benchmarks: ImageNet CUB200 EuroSAT Describable Textures Places365 Food101 Caltech101 Oxford Pets ImageNetV2 SUN397 Cars Flowers102
Metrics: accuracy
Baselines: Vanilla CLIP VCD Tip-Adapter TIP-X
Builds on: CLIP Tip-Adapter VCD CUPL
Closest papers by meaning
- Alpha-CLIP: A CLIP Model Focusing on Wherever You Want (CVPR 2024 · similarity 0.85)
- NeighborRetr: Balancing Hub Centrality in Cross-Modal Retrieval (CVPR 2025 · similarity 0.81)
- Interpreting CLIP's Image Representation via Text-Based Decomposition (ICLR 2024 · similarity 0.81)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.80)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.80)
- Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling (ICLR 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.