Research map / Vision-language and multimodal / CLIP and zero-shot
Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies spurious attributes in VLM attribute pools (SAP, using MLLM querying plus CBM weight thresholding) and proposes SAS, a plug-and-play pseudo-category loss that suppresses them within PEFT methods, improving few-shot out-of-distribution generalization on CLIP.
Problem
Few-shot adaptation of vision-language models trades off in-distribution accuracy against out-of-distribution generalization, and attribute-based methods rely on attribute pools that contain spuriously correlated attributes which hurt generalization.
Approach
Separate core from spurious attributes by prompting an MLLM, use concept bottleneck weights to flag high-influence non-core attributes as spurious, and then add a subsidiary classification loss in which pseudo categories built from spurious attributes (via Stable Diffusion or LAION retrieval) are contrasted with real categories.
Key results
- On base-to-new generalization, removing spurious attributes raised average new-category accuracy from 65.30% to 67.66% for CPL and from 66.07% to 67.69% for ArGue.
- SAS added to PromptSRC on ImageNet base-to-new raised accuracy from 74.01% to 75.46% (+1.45), with SAS shown to yield gains across most baselines averaging over 2%.
- SAS with CoCoOp on ImageNet-based cross-domain settings and group robustness: SAS achieved the best worst-group accuracy on 3 of 4 group-robustness datasets (e.g., 87.4% WG on CelebA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove spurious attributes from CPL's attribute pool (ablation) | CPL · Base-to-new, average across 11 datasets | new-category accuracy: 65.30% → 67.66% | Helped |
| Remove spurious attributes from ArGue's attribute pool (ablation) | ArGue · Base-to-new, average across 11 datasets | new-category accuracy: 66.07% → 67.69% | Helped |
| Modify attribute-generation prompts to focus on the category itself (ablation) | ArGue · Base-to-new, average across 11 datasets | new-category accuracy: 66.07% → 66.33% | Helped |
| Add SAS to CoCoOp | CoCoOp · ImageNet base-to-new | accuracy: 73.10 → 74.21 | Helped |
| Add SAS to PromptSRC | PromptSRC · ImageNet base-to-new | accuracy: 74.01 → 75.46 | Helped |
| Use pseudo categories based on spurious attributes rather than pseudo categories mirroring main categories (ablation) | SAS training · Base-to-new, average across 11 datasets | harmonic mean accuracy: 73.53% vs 76.36% | Helped |
| Use spurious-attribute pseudo categories instead of extending main-category data to 32 shots (ablation) | SAS training · Base-to-new, average across 11 datasets | harmonic mean accuracy: 74.34% vs 76.36% | Helped |
| Set SAS loss weight λ to 2 rather than λ=0 (ablation) | SAS · Base-to-new, average across 11 datasets | harmonic mean accuracy: 78.80 → 80.38 | Helped |
| Increase SAS loss weight from λ=2 to λ=20 (ablation) | SAS · Base-to-new, average across 11 datasets | harmonic mean accuracy: 80.38 → 79.14 | Hurt |
| Reduce image-generation diffusion steps from 100 to 50 (ablation) | CoCoOp + SAS · Base-to-new, average across 11 datasets | new-category accuracy: 67.09 → 66.97 | Hurt |
| Reduce image-generation diffusion steps from 50 to 25 (ablation) | CoCoOp + SAS · Base-to-new, average across 11 datasets | new-category accuracy: 66.97 → 66.05 | Hurt |
Surprising findings
- Synthetic Stable Diffusion images for pseudo categories outperformed real LAION-5B retrieved images for most baselines.
- Removing spurious attributes from zero-shot CLIP-based attribute prompts nearly reverted DCLIP to vanilla CLIP performance, suggesting spurious attributes drive much of the attribute gain.
Limitations
- SAP depends on GPT-4V (proprietary) and open-source MLLMs perform worse (BLIP-2 lowest), so identification quality depends on MLLM choice.
- Spurious attribute identification relies on a threshold/adaptive rule and can yield false positives that introduce noisy pseudo categories.
- Full SAS adds training cost (e.g., CoCoOp ImageNet 4h37m to 6h18m), and diffusion generation adds up to 3h50m for ImageNet.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Metrics: accuracy harmonic mean worst-group accuracy average-group accuracy
Baselines: CoCoOp TCP KgCoOp LASP PromptSRC CPL ArGue MAP MaPLe CLIP-Adapter Tip-Adapter RoboShot
Builds on: CoCoOp CPL ArGue PromptSRC Sus-X
Closest papers by meaning
- ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models (CVPR 2024 · similarity 0.90)
- RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations? (NeurIPS 2025 · similarity 0.89)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.89)
- Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models (ICML 2024 · similarity 0.88)
- Tree of Attributes Prompt Learning for Vision-Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.