Research map / Vision-language and multimodal / CLIP and zero-shot
LABEL-FREE MITIGATION OF SPURIOUS CORRELATIONS IN VLMS USING SPARSE AUTOENCODERS
Bharat Chandra Yalavarthi, Nalini K. Ratha, Venu Govindaraju
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add candidate selection (apply removal only to candidate samples) (ablation) | DIAL+ · Waterbirds ViT-L/14 | avg/worst-group accuracy: 81.60/51.56 -> 82.25/69.18 | Improved |
| add candidate selection (ablation) | DIAL+ · CelebA ViT-L/14 | avg/worst-group accuracy: 86.43/84.82 -> 86.54/85.15 | Improved |
| add candidate selection (ablation) | DIAL+ · CelebA ViT-B/32 | avg/worst-group accuracy: 84.85/81.67 -> 85.28/83.42 | Improved |
| add candidate selection (ablation) | DIAL+ · Waterbirds ViT-B/32 | avg/worst-group accuracy: 70.45/50.31 -> 71.88/52.82 | Improved |
| add candidate selection (ablation) | DIAL+ · FMOW ViT-L/14 | avg/worst-group accuracy: 26.04/19.55 -> 26.09/19.90 | No clear effect |
| use both mean activation difference and cosine similarity instead of cosine similarity alone (ablation) | DIAL · Waterbirds ViT-L/14 | avg/worst-group accuracy: 74.95/50.29 -> 82.6/68.69 | Improved |
| use both attribution terms instead of mean activation difference alone (ablation) | DIAL · Waterbirds ViT-L/14 | avg/worst-group accuracy: 82.61/68.85 -> 82.6/68.69 | No clear effect |
| use MSAE instead of BatchTopKSAE (ablation) | DIAL · Waterbirds | avg/worst-group accuracy: 71.98/41.51 -> 82.6/68.69 | Improved |
| use MSAE instead of JumpReLUSAE (ablation) | DIAL · COVID-19 | avg/worst-group accuracy: 58.33/34.48 -> 61.11/48.28 | Improved |
| use MSAE instead of JumpReLUSAE (ablation) | DIAL · CelebA | avg/worst-group accuracy: 87.74/84.44 -> 86.87/85.24 | Mixed |
| select features by attribution mass alpha instead of fixed top-k (ablation) | DIAL · CelebA, Waterbirds, FMOW (ViT-L/14) | accuracy | Improved |
| remove spurious subspace by orthogonal projection instead of zeroing neuron activations (ablation) | DIAL · CelebA, Waterbirds, FMOW (ViT-L/14) | accuracy | Improved |
Closest papers by meaning
- RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition (ICLR 2025 · similarity 0.91)
- Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations? (NeurIPS 2025 · similarity 0.91)
- Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.89)
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.89)
- Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
