Research map / Vision-language and multimodal / CLIP and zero-shot
Language-Driven Cross-Modal Classifier for Zero-Shot Multi-Label Image Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a linear multi-label classifier on CLIP text embeddings of 40K LLM-generated sentences, then transfers it to images via a label-specific cross-modal mapping fused with the original image embedding.
Problem
Zero-shot multi-label image recognition currently relies on annotated multi-label image data (seen classes) or few-shot tuning, which is costly; the modality gap between CLIP image and text embeddings also hurts transfer of text-trained classifiers.
Approach
Train a classifier purely on LLM-generated multi-label text embeddings from frozen CLIP, then map image embeddings into the text embedding space as softmax-weighted combinations of class text embeddings, fuse with the original embedding, and aggregate global and local region scores.
Key results
- Zero-shot CoMC reaches 68.7 mAP on MS-COCO, 89.4 on VOC2007 and 48.2 on NUS-WIDE, beating TaI-DPT by 3.6, 1.1 and 1.7 points.
- On NUS-WIDE, CoMC achieves 41.2 mAP versus DualCoOp's 41.3 (top-3 F1 comparison), and 48.2 F1 at top-5 vs DualCoOp's 43.6, with no image training.
- Removing the cross-modal mapping drops performance to 67.9 / 87.5 / 45.1 mAP on COCO / VOC / NUS-WIDE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-modal mapping (ablation) | CoMC · MS-COCO | mAP (%): 67.9 -> 68.7 | Helped |
| add cross-modal mapping (ablation) | CoMC · VOC2007 | mAP (%): 87.5 -> 89.4 | Helped |
| add cross-modal mapping (ablation) | CoMC · NUS-WIDE | mAP (%): 45.1 -> 48.2 | Helped |
| add local image embeddings (ablation) | CoMC · MS-COCO | mAP (%): 66.2 -> 68.7 | Helped |
| add local image embeddings (ablation) | CoMC · VOC2007 | mAP (%): 88.5 -> 89.4 | Helped |
| add local image embeddings (ablation) | CoMC · NUS-WIDE | mAP (%): 47.6 -> 48.2 | Helped |
| use GPT-3 instead of ChatGLM-6b to generate text data (ablation) | CoMC · MS-COCO | mAP (%): 66.2 -> 68.7 | Helped |
| use GPT-3 instead of ChatGLM-6b to generate text data (ablation) | CoMC · VOC2007 | mAP (%): 85.1 -> 89.4 | Helped |
| use GPT-3 instead of ChatGLM-6b to generate text data (ablation) | CoMC · NUS-WIDE | mAP (%): 42.8 -> 48.2 | Helped |
| use GPT-3 instead of Llama-2-7b to generate text data (ablation) | CoMC · MS-COCO | mAP (%): 66.1 -> 68.7 | Helped |
| use GPT-3 instead of Llama-2-7b to generate text data (ablation) | CoMC · VOC2007 | mAP (%): 87.5 -> 89.4 | Helped |
| use GPT-3 instead of Llama-2-7b to generate text data (ablation) | CoMC · NUS-WIDE | mAP (%): 46.9 -> 48.2 | Helped |
Surprising findings
- Zero-shot CoMC with no image training outperforms 4-shot CoOp-DPT on VOC2007 and 8-shot CoOp-DPT on MS-COCO.
Limitations
- Mapping relies on simple text templates; performance drops sharply when the fusion weight alpha is below 0.4.
- Performance depends on LLM quality, with GPT-3 outperforming smaller open LLMs (ChatGLM-6B, Llama-2-7B).
Experimental setup
Datasets: MS-COCO VOC2007 NUS-WIDE GPT-3 generated text (40K sentences)
Benchmarks: MS-COCO VOC2007 NUS-WIDE
Metrics: mAP precision at top-K recall at top-K F1 at top-K
Baselines: Zero-shot CLIP CLIP-DPT TaI-DPT CONSE LabelEM Fast0Tag One Attention per Label LESA BiAM SDL MKT DualCoOp
Builds on: CLIP TaI-DPT DualCoOp CoOp
Closest papers by meaning
- Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfer (AAAI 2023 · similarity 0.90)
- Cross-Modal Representation Learning for Zero-Shot Action Recognition (CVPR 2022 · similarity 0.88)
- [CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation (ICML 2026 · similarity 0.88)
- SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models (CVPR 2025 · similarity 0.88)
- Label Propagation for Zero-shot Classification with Vision-Language Models (CVPR 2024 · similarity 0.87)
- MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.