Research map / Vision-language and multimodal / CLIP and zero-shot

Multimodal Ensembling for Zero-Shot Image Classification

AAAI 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Attribute-based zero-shot learning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a zero-shot multimodal ensemble that combines CuPL text-prompt CLIP classification with image-to-image matching against Stable Diffusion XL generated class images, as a proposal without reported experiments.

Problem

Multimodal image classification, especially fine-grained classification, struggles with underfitting in low-capacity models and with obtaining high-quality cross-modal data for each class, while needing zero-shot generalization to unseen classes.

Approach

Ensemble CLIP text-to-image scores (using CuPL descriptive prompts) with CLIP image-to-image scores against Stable Diffusion XL generated class images (with negative prompts) to get a zero-shot classifier requiring no training or fine-tuning.

Key results

• No experimental results are reported; the paper is a proposal describing planned evaluation against the CLIP top-1 accuracy baseline.
• The authors hypothesize that the method will beat NTS-Net top-1 accuracy on a dataset neither was trained on, but do not report this test.

Limitations

• No experiments or quantitative results are reported; the work is a proposal.
• Method assumes the generative model represents the data distribution better and that the base model can distinguish similar classes.
• The method offsets learned biases rather than ensuring fairness.

Experimental setup

Datasets: ImageNet DTD FGVC Aircraft Flowers 102

Benchmarks: ImageNet DTD FGVC Aircraft Flowers 102

Metrics: top-1 accuracy precision recall F1 score

Baselines: CLIP ViT-L/14 top-1 accuracy NTS-Net

Builds on: CuPL CLIP Stable Diffusion XL bagging NTS-Net

Closest papers by meaning

• Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.88)
• Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs (NeurIPS 2025 · similarity 0.86)
• No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance (NeurIPS 2024 · similarity 0.86)
• With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.86)
• Multimodal Representation Learning by Alternating Unimodal Adaptation (CVPR 2024 · similarity 0.85)
• Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.85)