Research map / Vision-language and multimodal / CLIP and zero-shot
SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Treats CLIP as a black box and improves zero-shot multi-label recognition by standardizing VLM scores and fusing compound-prompt order statistics via max-variance (PCA) rank fusion.
Problem
Zero-shot multi-label recognition with vision-language models suffers from image- and prompt-specific score biases and AND/OR ambiguity in compound prompts, hurting mAP without training data or model modification.
Approach
Score-only black-box method: standardize CLIP scores at image and prompt levels, build compound prompts from class co-occurrence, and fuse order statistics of compound scores (not the max) with the singleton score using a max-variance projection, showing the second-highest compound score is a better signal than the highest.
Key results
- Improves vanilla ZSCLIP by 12.6% mAP on COCO, 8.8% on VOC, and 7.9% on NUS-WIDE averaged over nine CLIP backbones.
- Adds 1.6% average gain on top of TagCLIP (COCO/VOC ViT-B/16 average 81.3 to 82.9) and 1.7% on top of TaI-DPT.
- Standardization alone improves mAP by 6-10% on COCO, VOC, and NUS-WIDE; second-highest compound score outperforms the maximum.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SPARC normalization, compound prompts, and adaptive fusion | ZSCLIP · COCO2014 test | mAP: 55.7 -> 68.3 | Helped |
| Apply SPARC normalization, compound prompts, and adaptive fusion | ZSCLIP · VOC2007 test | mAP: 80.4 -> 89.2 | Helped |
| Apply SPARC normalization, compound prompts, and adaptive fusion | ZSCLIP · NUSWIDE test | mAP: 39.3 -> 47.2 | Helped |
| Apply SPARC to TagCLIP scores | TagCLIP (ViT-B/16) · COCO | mAP: 70.9 -> 73.8 | Helped |
| Apply SPARC to TagCLIP scores | TagCLIP (ViT-B/16) · VOC | mAP: 91.7 -> 92.0 | Helped |
| Apply SPARC to TaI-DPT scores | TaI-DPT (RN50) · COCO | mAP: 65.1 -> 68.2 | Helped |
| Apply SPARC to TaI-DPT scores | TaI-DPT (RN50) · VOC | mAP: 88.5 -> 90.3 | Helped |
| Apply SPARC to TaI-DPT scores | TaI-DPT (RN50) · NUSWIDE | mAP: 46.2 -> 46.6 | Helped |
| Apply SPARC to CoMC scores | CoMC (RN50) · COCO | mAP: 68.8 -> 68.9 | Helped |
| Add normalization to singleton prompts without compounds (ablation) | ZSCLIP · COCO | mAP: 55.7 -> 65.9 | Helped |
| Add normalization to singleton prompts without compounds (ablation) | ZSCLIP · VOC | mAP: 80.4 -> 87.7 | Helped |
| Add normalization to singleton prompts without compounds (ablation) | ZSCLIP · NUSWIDE | mAP: 39.3 -> 45.1 | Helped |
Surprising findings
- The highest-scoring compound prompt is a poor detection signal; lower-order statistics (2nd, 3rd highest) separate positives better due to OR-like VLM scoring.
- Randomized WaffleCLIP-style prompts add no complementary signal, suggesting gains come from compound prompt semantics.
Limitations
- Relies on ground-truth co-occurrence statistics from the training set to filter compound prompts (though ablation shows gains without filtering).
- Gains over CoMC are not achieved (slight degradation of 0.3%).
Experimental setup
Datasets: COCO2014 VOC2007 NUS-WIDE
Benchmarks: COCO2014 VOC2007 NUS-WIDE
Metrics: mAP
Baselines: ZSCLIP CLIP-DPT TagCLIP TaI-DPT CoMC CLIP-Surgery CDUL
Builds on: CLIP ZSCLIP prompt ensembling TaI-DPT TagCLIP CoMC
Closest papers by meaning
- Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.88)
- Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting (NeurIPS 2024 · similarity 0.88)
- Language-Driven Cross-Modal Classifier for Zero-Shot Multi-Label Image Recognition (ICML 2024 · similarity 0.88)
- Making Large Vision Language Models to Be Good Few-Shot Learners (AAAI 2025 · similarity 0.87)
- An Empirical Study Into What Matters for Calibrating Vision-Language Models (ICML 2024 · similarity 0.87)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.