Research map / Vision-language and multimodal / CLIP and zero-shot
Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEraser, a test-time prompt tuning method that minimizes KL divergence to uniform predictions on auxiliary "spurious" images so CLIP learns to ignore decision shortcuts, plus a new shortcut-based evaluation protocol (S2E).
Problem
Vision-language foundation models like CLIP rely on decision shortcuts (e.g., background cues) that hurt zero-shot performance on fine-grained or shifted tasks, and existing robustness benchmarks may be unsuitable for them.
Approach
Instead of telling the model which features are invariant, optimize a learnable prompt at test time so that predictions on auxiliary images containing spurious features (background annotations, image patches, or reference images) become uniform, thereby erasing reliance on shortcuts without changing model weights.
Key results
- On Waterbirds, SEraser raises worst-group accuracy from 40.04 (Vanilla CLIP ViT-B/32) to 65.25 (+25.21).
- Averaged over the simulated S2E scenarios and Waterbirds, worst-group accuracy improves from 49.48 to 83.72 (+34.24).
- On real-world datasets, the combined Both strategy averages 40.5 vs 25.8 for Vanilla CLIP, though Alpha-CLIP scores 57.3 there, which the authors attribute to ImageNet leakage in its training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use test-time prompt tuning with background annotations instead of zero-shot Vanilla | CLIP ViT-B-32 · Waterbirds | worst-group accuracy: 40.04 -> 65.25 | Helped |
| Use test-time prompt tuning with background annotations instead of zero-shot Vanilla | CLIP ViT-B-32 · CamelDeer | worst-group accuracy: 66.40 -> 91.60 | Helped |
| Use test-time prompt tuning with background annotations instead of zero-shot Vanilla | CLIP ViT-B-32 · SpiderCrab | worst-group accuracy: 42.00 -> 94.67 | Helped |
| Use patches and reference images together as auxiliary images (ablation) | SEraser · Tiny-ImageNet | accuracy: 23.2 -> 42.8 | Helped |
| Use patches and reference images together as auxiliary images (ablation) | SEraser · CUB-200 | accuracy: 12.1 -> 28.9 | Helped |
| Use patches and reference images together as auxiliary images (ablation) | SEraser · ImageNetA | accuracy: 42.1 -> 49.7 | Helped |
| Replace corner patches with randomly sampled patches (ablation) | SEraser · real-world scenarios | accuracy | Helped |
| Apply test-time prompt tuning | BLIP-2 · Waterbirds | average accuracy: 57.65 -> 55.56 | Hurt |
Surprising findings
- TPT underperforms vanilla CLIP on Waterbirds worst-group accuracy (34.38 vs 40.04), since its confidence-based view selection can favor spurious backgrounds.
- The method still helps when auxiliary patches contain invariant features (random patch sampling), which the authors argue acts as a soft constraint.
Limitations
- Relies on auxiliary images or spurious-feature annotations (e.g., SAM background masks), whose quality affects results.
- Evaluation on S2E-generated datasets uses only subsets due to file size limits; Alpha-CLIP comparison on Tiny-ImageNet is confounded by ImageNet training data.
Experimental setup
Datasets: Tiny-ImageNet CUB-200 ImageNet-A Waterbirds CamelDeer (S2E-generated) SpiderCrab (S2E-generated)
Benchmarks: Tiny-ImageNet CUB-200 ImageNet-A Waterbirds CamelDeer SpiderCrab
Metrics: accuracy (average) worst-group accuracy
Baselines: Vanilla CLIP MASK TPT RoSHOT Alpha-CLIP
Builds on: test-time prompt tuning (TPT) CLIP Segment Anything
Closest papers by meaning
- Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.89)
- FATE: Feature-Adapted Parameter Tuning for Vision-Language Models (AAAI 2025 · similarity 0.88)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.88)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation (NeurIPS 2025 · similarity 0.88)
- BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.