Research map / Vision-language and multimodal / CLIP and zero-shot
Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoSEP, a black-box, fully unsupervised iterative prompt-optimization framework that learns an image-description-generation prompt for MLLMs using an instance-level description-retrieval score on unlabeled images, improving fine-grained zero-shot classification.
Problem
MLLMs underperform on fine-grained image classification of visually similar subcategories, and prior fixes need large labeled datasets or white-box training; the paper asks whether unlabeled test-time images can improve MLLM zero-shot fine-grained classification under black-box access.
Approach
Insert an intermediate image-description step whose generation prompt is optimized with an unsupervised instance-level classification score (the MLLM must retrieve each image's own description among negatives), using Reflect and Modify operators in an iterative beam-like search.
Key results
- AutoSEP averages about 13% improvement over vanilla zero-shot and about 3% over the best baseline across 8 fine-grained tasks and three MLLMs.
- With GPT-4o on CUB_cuckoo, AutoSEP reaches 75.12% vs 61.46% vanilla zero-shot.
- With Qwen2-VL-7B-Instruct (hybrid setup), AutoSEP reaches 61.79% on CUB_cuckoo vs 40.24% vanilla zero-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize the description-generation prompt using unlabeled data | Gemini 1.5 Flash · Fine-grained classification tasks | accuracy | Helped |
| Use generated descriptions from an initial human-crafted prompt | Zero-shot image classification · Fine-grained classification tasks | accuracy | Mixed |
| Replace random image sampling with same-class-only sampling (ablation) | AutoSEP · CUB_cuckoo | accuracy: 61.22 -> 61.58 | No clear effect |
| Replace random image sampling with different-class-only sampling (ablation) | AutoSEP · CUB_cuckoo | accuracy: 61.22 -> 60.98 | No clear effect |
| Increase the number of unlabeled samples used for optimization (ablation) | AutoSEP · Fine-grained classification tasks | accuracy | Helped |
| Add more optimization iterations (ablation) | AutoSEP · Fine-grained classification tasks | accuracy | Helped |
| Use instance-level classification accuracy as the optimization signal instead of random-label, majority-vote, or SPO scoring (ablation) | Prompt optimization for fine-grained image classification · Fine-grained classification tasks | accuracy | Helped |
Surprising findings
- SPO, a strong text-only self-supervised prompt optimizer, transfers poorly to fine-grained image classification, as MLLMs struggle to judge description informativeness.
- Sampling same-class images for instance-level classification performs about as well as random or different-class sampling.
Limitations
- Assumes discriminative visual features can be verbalized in natural language.
- Requires a sufficient number of unlabeled samples for stable optimization and still underperforms fully supervised methods.
- Relies on a strong MLLM agent for Reflect and Modify; smaller models need a hybrid setup with a stronger model.
Experimental setup
Datasets: CUB-200-2011 iNaturalist 2021 Stanford Dogs VegFru
Benchmarks: CUB_cuckoo CUB_oriole CUB_vireo iNat_butterfly iNat_lupine StanfordDogs_terrier VegFru_greens VegFru_allium
Metrics: accuracy
Baselines: Vanilla zero-shot Zero-shot with descriptions Zero-shot with majority vote Few-shot with random labels Few-shot with MLLM labels Multiple images display K-means clustering SPO
Builds on: SPO automatic prompt engineering reflect-and-modify contrastive self-supervised learning
Closest papers by meaning
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.88)
- Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.87)
- ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.86)
- Multimodal Ensembling for Zero-Shot Image Classification (AAAI 2024 · similarity 0.86)
- Zero-Shot Robustification of Zero-Shot Models (ICLR 2024 · similarity 0.86)
- Multi-Label Few-Shot Image Classification via Pairwise Feature Augmentation and Flexible Prompt Learning (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.