Research map / Vision-language and multimodal / CLIP and zero-shot
Boosting the visual interpretability of CLIP via adversarial fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reproduces the ICLR 2025 unsupervised adversarial fine-tuning (AFT) study that improves CLIP ViT-B/16 visual interpretability, and extends it with an Energy Pointing Game loss for saliency-guided regularisation on COCO.
Problem
Gradient-based saliency maps from CLIP's image encoder are noisy and diffuse, and the encoder has few interpretable concept detectors; the original work claims AFT fixes this at a modest accuracy cost. This study checks those claims and tests whether explicit saliency supervision improves localisati
Approach
Reimplements unsupervised AFT with norm regularisation, adds an Energy Pointing Game loss that penalises Simple Gradient saliency mass falling outside COCO foreground masks, and sweeps the loss weight to expose a localisation vs. accuracy trade-off.
Key results
- Under AFT^4 Simple Gradient Pointing Game on ImageNet-S rises from 25.26 to 56.55 (zero-shot), and network dissection detectors rise from 160 to 390 of 768 neurons.
- EPG loss with w_s=0.6 raises Simple Gradient PG on ISIC from 0.428 to 0.820 and COCO-probe PG from 0.333 to 0.467 relative to AFT^4.
- At eps=1/255 the average clean zero-shot accuracy drops 7.1 points vs CLIP, driven by EuroSAT where all AFT models reach chance; natural-image drops are at most about 3 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply adversarial fine-tuning with ε=4/255 | CLIP ViT-B/16 · ImageNet-Segmentation validation, zero-shot | Simple Gradient PG: 25.26 -> 56.55 | Helped |
| apply adversarial fine-tuning with ε=4/255 | CLIP ViT-B/16 · Broden | Soft-WPMI mean: 0.2264 -> 0.2817 | Helped |
| apply adversarial fine-tuning with ε=4/255 | CLIP ViT-B/16 · Broden, IoU ≥ 0.04 | concept detectors: 160 to 390 | Helped |
| apply adversarial fine-tuning with ε=4/255 | CLIP ViT-B/16 · ISIC 2024, zero-shot | Simple Gradient PG: 0.107 to 0.424 | Helped |
| apply adversarial fine-tuning with ε=4/255 | CLIP ViT-B/16 · Kvasir-SEG, zero-shot | Simple Gradient PG: 0.058 -> 0.056 | No clear effect |
| apply adversarial fine-tuning with ε=4/255 | CLIP ViT-B/16 · COCO 6-class linear probe | Simple Gradient PG: 0.140 to 0.333 | Helped |
| apply adversarial fine-tuning with ε=4/255 | CLIP ViT-B/16 · CIFAR-10, clean | zero-shot accuracy: 89.4 -> 66.2 | Hurt |
| apply adversarial fine-tuning with ε=4/255 | CLIP ViT-B/16 · CIFAR-10, 2/255 attack | zero-shot accuracy: 0.0 -> 44.9 | Helped |
| apply adversarial fine-tuning with ε=1/255 | CLIP image encoder · Flickr30k, clean, text-to-image | Recall@1: 24.95 -> 27.20 | Helped |
| apply adversarial fine-tuning with ε=1/255 | CLIP image encoder · Flickr30k, clean, image-to-text | Recall@1: 44.41 to 43.02 | Hurt |
| add EPG loss with w_s=0.6 (ablation) | AFT^4 · ISIC 2024, zero-shot | Simple Gradient PG: 0.428 to 0.820 | Helped |
| add EPG loss with w_s=0.6 (ablation) | AFT^4 · Kvasir-SEG, zero-shot | Simple Gradient PG: 0.056 to 0.265 | Helped |
Surprising findings
- Grad-CAM is unstable on CLIP ViT-B/16 (fragmented blocky maps) though it works on ResNet-50, so the Grad-CAM half of the claim was inconclusive.
- Absolute localisation and network dissection counts differ from the original paper, shifting mostly with undocumented protocol choices such as the IoU threshold (detector count varies ninefold across IoU 0.02-0.06).
Limitations
- Absolute metric values depend on undocumented evaluation-protocol choices and do not numerically match the original paper.
- The EPG extension requires dense instance masks (COCO-only), and the EPG models are fine-tuned on COCO while the AFT baseline is ImageNet-only, confounding part of the comparison.
Experimental setup
Datasets: ImageNet 2012 COCO 2017 ImageNet-S Broden Caltech256 Imagenette Flickr30k ISIC 2024 Chest X-ray Wildlife Animals Kvasir-SEG CIFAR-10
Benchmarks: ImageNet-S Broden CIFAR-10 STL-10 EuroSAT FGVC Aircraft Imagenette Flickr30k Caltech256 ISIC 2024 Kvasir-SEG COCO linear probe
Metrics: Pointing Game Energy Pointing Game Average Precision Pixel Accuracy IoU zero-shot accuracy Recall@k ROAR accuracy detector count (IoU threshold) Soft-WPMI
Baselines: CLIP (w/o AFT) AFT^4 (original checkpoint) AFT^1 COCO AFT (w_s=0) Grad-ECLIP Grad-CAM
Builds on: CLIP adversarial fine-tuning (AFT) PGD APGD / AutoAttack CLIP-Dissect ROAR
Closest papers by meaning
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.88)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.88)
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.88)
- Enhancing CLIP Robustness via Cross-Modality Alignment (NeurIPS 2025 · similarity 0.87)
- Gradient-based Visual Explanation for Transformer-based CLIP (ICML 2024 · similarity 0.86)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.