Research map / Vision-language and multimodal / Multimodal LLMs
Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes zero-shot prompt ensembling (ZPE), which scores a large pool of prompt templates without labels using bias-corrected max-logit scores and softmax weighting, to build weighted prompt ensembles for CLIP and LiT zero-shot classifiers.
Problem
Zero-shot classifiers of contrastive text-image models need hand-crafted prompt engineering, which is labor-intensive and usually requires labeled validation data; the goal is to automate prompt selection and weighting without labels.
Approach
Score each prompt by its average max logit over test images, correct frequency biases by subtracting expected logits over pre-training (LAION-400M subsample) and test images, and apply softmax weighting (or outlier-based selection via MAD z-scores) to form a weighted prompt ensemble.
Key results
- For CLIP ViT-B/16 averaged over 11 fine-grained datasets, ImageNet and its 4 variants, ZPE weighted average reaches 67.44% vs 66.06% for the equal-average pool set (hand-crafted equal average 67.29%).
- For LiT ViT-L/16 across all datasets, ZPE prompt selection reaches 77.38% vs 74.74% equal-average pool set and 76.51% hand-crafted.
- Normalization by E_pretrain and E_test yields +0.54% average accuracy over no normalization for the weighted average on CLIP ViT-B/16 (ImageNet-based ablation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use bias-normalized prompt selection rather than equal averaging of hand-crafted prompts | CLIP ViT-B/16 zero-shot classification · ImageNet and variants (Avg) | accuracy: 60.85 -> 61.21 | Helped |
| Use bias-normalized prompt selection rather than equal averaging of hand-crafted prompts | LiT ViT-L/16 zero-shot classification · Fine-grained classification (Avg) | accuracy: 77.73 -> 78.71 | Helped |
| Combine pre-training and test-image expectation normalization instead of no normalization (ablation) | CLIP ViT-B/16 ZPE weighted average · All datasets (Avg) | accuracy: 66.92 -> 67.44 | Helped |
| Use pre-training expectation normalization rather than no normalization (ablation) | CLIP ViT-B/16 ZPE weighted average · All datasets (Avg) | accuracy: 66.92 -> 67.42 | Helped |
| Use test-image expectation normalization alone rather than no normalization (ablation) | CLIP ViT-B/16 ZPE weighted average · All datasets (Avg) | accuracy: 66.92 -> 67.00 | No clear effect |
| Take expectation over images and classes rather than images only for pre-training normalization (ablation) | CLIP ViT-B/16 ZPE weighted average · All datasets (Avg) | accuracy: 67.42 -> 67.17 | Hurt |
| Use softmax weighting rather than raw prompt scores (ablation) | CLIP ViT-B/16 ZPE weighted average · All datasets (Avg) | accuracy: 66.18 -> 67.44 | Helped |
| Raise prompt scores to the power of 10 rather than using raw scores (ablation) | CLIP ViT-B/16 ZPE weighted average · All datasets (Avg) | accuracy: 66.18 -> 67.30 | Helped |
| Expand the pool from 247 to 426 prompts with generated templates (ablation) | CLIP ViT-B/16 ZPE weighted average · All datasets (Avg) | accuracy: 67.44 -> 67.30 | Hurt |
| Estimate pre-training expectation with 5k rather than 20k random images (ablation) | CLIP ViT-B/16 ZPE weighted average · All datasets (Avg) | accuracy: 67.44 -> 67.44 | No clear effect |
| Estimate test-image expectation using 10% rather than 100% of test images (ablation) | CLIP ViT-B/16 ZPE weighted average · All datasets (Avg) | accuracy: 67.44 -> 67.44 | No clear effect |
| Use per-example rather than per-dataset prompt scoring (ablation) | CLIP ViT-B/16 ZPE with pool prompts · All datasets (Avg) | accuracy: 67.44 -> 67.60 | Helped |
Surprising findings
- For CIFAR10 and EuroSat, equal weighting of all pool prompts was the best performing method.
- Naive max-logit prompt scores are dominated by spurious prompts containing the word 'person', reflecting word frequency bias in pre-training data and spurious concept bias in test data.
Limitations
- Requires a large, varied pool of high-quality prompt templates, which was not available and had to be simulated.
- Prompts are scored independently, ignoring combinations of prompts.
- Scoring is per dataset rather than per image; per-example scoring is only an initial investigation.
Experimental setup
Datasets: LAION-400M (20k subsample for pre-training reference) 27 prompt sets from CLIP prompts for 14 datasets from CLIP ChatGPT-generated prompt templates
Benchmarks: ImageNet ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-V2 Caltech101 Cars196 CIFAR10 CIFAR100 DTD EuroSat Food-101
Metrics: zero-shot top-1 accuracy
Baselines: hand-crafted prompts equal average pool set equal average max-logit scoring class name only A photo of {} TPT
Builds on: CLIP prompt ensembling test-time prompt tuning CoOp
Closest papers by meaning
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.90)
- Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage (ICML 2025 · similarity 0.86)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.86)
- BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs (CVPR 2025 · similarity 0.86)
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.86)
- Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.