Research map / Vision-language and multimodal / CLIP and zero-shot
PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PerceptionCLIP, a training-free two-step zero-shot CLIP classifier that first infers contextual attributes (e.g., background, orientation) from the image and then classifies conditioned on them.
Problem
Zero-shot image classification with CLIP treats classification as text retrieval with simple or ad-hoc prompt templates, ignoring contextual attributes like background and orientation, which leads to sub-optimal generalization and reliance on spurious features.
Approach
Model images as generated from the class plus contextual attributes, define an attribute-aware CLIP score over descriptions of class and attributes, and marginalize over self-inferred attribute distributions to classify, emulating human perception; prompt ensembling is shown to be a special case.
Key results
- On ImageNet-Sketch with ViT-B/16, using the top-4 contextual attributes reaches 49.10% vs 46.16% for a single template and 48.26% for 80-template ensembling.
- On EuroSAT with ViT-B/16, PerceptionCLIP reaches 59.23% vs 51.44% for a simple template (near 5% gain with image source and condition attributes).
- On Waterbirds with ViT-L/14, the worst-group-to-average gap falls from 45.91 to 26.62 points using background and background+ attributes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Infer and condition on the top 4 contextual attributes instead of using 80 templates | CLIP ViT-B/16 · ImageNet-R | accuracy: 77.71 -> 78.38 | Helped |
| Add contextual attributes, including image source and condition | CLIP ViT-B/16 · EuroSAT | accuracy: 54.94 -> 59.23 | Helped |
| Condition classification on ground-truth contextual attributes rather than omitting attributes | CLIP · ImageNet, transformed-image average | accuracy: +1.16 | Helped |
| Condition classification on randomly generated contextual attributes instead of omitting attributes (ablation) | CLIP · ImageNet, transformed-image average | accuracy: +0.64 | Mixed |
| Condition classification on wrong contextual attribute values instead of omitting attributes (ablation) | CLIP · ImageNet, transformed-image average | accuracy: +0.57 | Mixed |
| Intervene in inferred attributes using temperature 3 and the top four attributes (ablation) | PerceptionCLIP · ImageNet | accuracy: 68.59% -> 68.72% | Helped |
| Condition on background+ rather than using no context | CLIP ViT-L/14 · Waterbirds | worst-group accuracy: 44.64 -> 61.12 | Helped |
| Condition on gender, age, and race rather than using no context | CLIP ViT-L/14 · CelebA | worst-group accuracy: 77.36 -> 80.84 | Helped |
| Use correctly matched contextual descriptions rather than no context | CLIP ViT-B/32 · Waterbirds | core-feature saliency: 62.8 -> 66.3 | Helped |
Surprising findings
- Conditioning on random character strings as attributes still yields about half the benefit of correct attributes, suggesting part of the gain comes from noise-like augmentation.
- Single-attribute conditioning on many attributes (e.g., background, quality) already beats the 80-template prompt ensemble on several ImageNet variants.
Limitations
- Sensitivity to text descriptions inherent to CLIP; using a distribution of descriptions only partly alleviates it.
- CLIP does not always infer contextual attributes correctly, so temperature tuning or human intervention may be needed.
Experimental setup
Datasets: ImageNet ImageNet-C Waterbirds CelebA
Benchmarks: ImageNet ImageNetV2 ImageNet-R ImageNet-A ImageNet-Sketch CUB200 EuroSAT Places365 Flowers102 Food101 Oxford Pets Waterbirds
Metrics: accuracy worst-group accuracy accuracy gap
Baselines: single template (a photo of a {class}) 80 prompt ensemble templates domain template
Builds on: CLIP prompt ensembling Grad-CAM Segment Anything
Closest papers by meaning
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.92)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.88)
- DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception (CVPR 2025 · similarity 0.88)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.87)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.87)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.