Research map / Vision-language and multimodal / Multimodal LLMs

[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Reproduces the ICLR 2025 unsupervised adversarial fine-tuning (AFT) paper that improves CLIP ViT-B/16 visual interpretability, and extends it with an Energy Pointing Game loss that supervises Simple Gradient saliency with COCO masks.

Problem

Gradient-based saliency maps of pretrained CLIP image encoders are noisy and diffuse, and the original AFT method does not explicitly enforce alignment of saliency with semantically meaningful object regions.

Approach

Independently reproduce AFT's four claims (saliency quality, concept alignment, transferability, accuracy trade-off), implement missing evaluation protocols, and add an EPG loss on Simple Gradient maps during COCO fine-tuning after an AFT checkpoint, swept over saliency weights.

Key results

• Under AFT^4, Simple Gradient pointing game on ImageNet-S rises from 25.26 to 56.55 (retrained) versus 56.61 for the original checkpoint.
• Network dissection detector count rises from 160 to 390 of 768 neurons for AFT^4 at IoU 0.04, matching the original checkpoint exactly; the absolute count is highly sensitive to IoU threshold.
• EPG loss at w_s=0.6 raises ISIC Simple Gradient PG from 0.428 to 0.820 and COCO-probe SG PG to 0.467, at a clean/adversarial accuracy cost of roughly 3-4 points versus the w_s=0 control.

Surprising findings

• Standard Grad-CAM remains unstable on CLIP ViT-B/16, while the same pipeline gives smooth maps on a ResNet-50, so the authors treat the Grad-CAM claim as inconclusive.
• AFT models score at chance (about 10%) on EuroSAT zero-shot, so the accuracy-trade-off claim holds only on natural-image benchmarks.

Limitations

• Absolute localisation values differ from the original paper due to undocumented evaluation protocol choices.
• The EPG loss requires dense instance masks (COCO), restricting training data compared with unsupervised AFT.
• Deletion/insertion and prompt-specific localisation show no consistent improvement from the saliency extension.

Experimental setup

Datasets: ImageNet 2012 COCO 2017 ImageNet-S Caltech256 Imagenette Broden Flickr30k ISIC 2024 Chest X-ray Wildlife Animals Kvasir-SEG CIFAR-10

Benchmarks: ImageNet-S Broden CIFAR-10 STL-10 EuroSAT FGVC Aircraft Imagenette Flickr30k Caltech256 ISIC 2024 Kvasir-SEG COCO

Metrics: Pointing Game Energy Pointing Game Average Precision Pixel Accuracy IoU Soft-WPMI cosine similarity Recall@k zero-shot accuracy robust accuracy under APGD/AutoAttack

Baselines: CLIP w/o AFT AFT^4 original checkpoint AFT^1 COCO AFT (w_s=0)

Builds on: adversarial fine-tuning (AFT) PGD APGD CLIP-Dissect Grad-ECLIP

Closest papers by meaning

• XFeat Revisited: Reproducibility and Evaluation of a Lightweight Image Matcher (NeurIPS 2026 · similarity 0.97)
• Revisiting "Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editing" (NeurIPS 2026 · similarity 0.96)
• Debate with Images: Detecting Deceptive Behaviors in Multimodal Large Language Models (ICML 2026 · similarity 0.93)
• DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarios (NeurIPS 2025 · similarity 0.88)
• The Coherence Trap: When MLLM-Crafted Narratives Exploit Manipulated Visual Contexts (CVPR 2026 · similarity 0.87)
• Among Us: A Sandbox for Measuring and Detecting Agentic Deception (NeurIPS 2025 · similarity 0.87)