Research map / Vision-language and multimodal / Multimodal LLMs
SLADE: Shielding against Dual Exploits in Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SLADE, an unsupervised adversarial fine-tuning of the CLIP ViT-L/14 vision encoder using symmetric patch-level and global stop-gradient contrastive losses, which is plugged into LVLMs to improve robustness against gradient-based and jailbreak attacks.
Problem
LVLMs with CLIP vision encoders are vulnerable to gradient-based adversarial perturbations and optimization-based jailbreak attacks; existing robust CLIP fine-tuning methods lose fine-grained semantic detail under attack or need large batches/momentum encoders.
Approach
Adversarially fine-tune CLIP without labels by combining a symmetric patch-level contrastive loss (aligning clean and adversarial patch embeddings) with a global image-level loss, both using stop-gradient cosine similarity so no large negative batches or momentum encoder are needed.
Key results
- SLADE4 on LLaVA-7B under APGD eps=4/255 yields COCO CIDEr 92.1 vs 13.5 for original CLIP and 75.6-ish for TeCoA4-level baselines (reported in Table 1 range).
- Targeted attacks at eps=4/255: SLADE4 achieves 0% attack success rate with average CIDEr 87.5, versus 100% ASR and CIDEr 0 for original CLIP.
- VisualAdv toxicity on LLaVA-13B drops from 30.4% (CLIP) to 13.7% with SLADE4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the default CLIP vision encoder with SLADE4 | OpenFlamingo · COCO, ℓ∞ 4/255 image captioning | CIDEr: 52.2 -> 62.1 | Helped |
| Replace the default CLIP vision encoder with SLADE4 | LLaVA-7B · COCO, ℓ∞ 4/255 image captioning | CIDEr: 83.5 -> 92.1 | Helped |
| Replace the default CLIP vision encoder with SLADE4 | CLIP ViT-L/14 zero-shot classifier · AutoAttack ℓ∞ 4/255 across six datasets | accuracy | Mixed |
| Replace the default CLIP vision encoder with SLADE4 | LLaVA-7B · COCO targeted attack, ℓ∞ 4/255 | CIDEr: 78.7 -> 87.5 | Helped |
| Replace the default CLIP vision encoder with SLADE4 | LLaVA-7B · COCO targeted attack, ℓ∞ 4/255 | attack success rate: 0% -> 0% | No clear effect |
| Replace the default CLIP vision encoder with SLADE4 | LLaVA-13B · VisualAdv jailbreak, ℓ∞ 128/255 | average toxicity: 30.4 -> 13.7 | Helped |
| Replace the default CLIP vision encoder with SLADE4 | LLaVA-13B · ImgJP jailbreak, ℓ∞ 16/255 | ASR: 28.0 ± 0.2 -> 15.0 ± 0.3 | Helped |
| Replace the default CLIP vision encoder with SLADE4 | LLaVA-13B · VisIT-Bench instruction following | BERTScore F1: 0.892 -> 0.889 | No clear effect |
| Replace the default CLIP vision encoder with SLADE2 | LLaVA-13B · Science Question Answering image-question pairs | accuracy: 64.5 -> 63.4 | No clear effect |
Limitations
- Clean performance is slightly degraded relative to original CLIP due to adversarial fine-tuning
- Evaluation limited to a few LVLMs (OpenFlamingo, LLaVA-1.5) and attacks
Experimental setup
Datasets: ImageNet COCO Flickr30k OKVQA VizWiz CIFAR10 CIFAR100 StanfordCars EuroSAT PCAM Flowers VisIT-Bench
Benchmarks: COCO Flickr30k OKVQA VizWiz CIFAR10 CIFAR100 StanfordCars Flowers EuroSAT PCAM VisIT-Bench ScienceQA
Metrics: CIDEr VQA accuracy robust accuracy attack success rate toxicity percentage BERTScore METEOR accuracy
Baselines: original CLIP TeCoA FARE JailGuard Diffpure
Builds on: TeCoA FARE SimSiam (stop-gradient) PGD adversarial training LLaVA-1.5
Closest papers by meaning
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.90)
- Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models (CVPR 2025 · similarity 0.90)
- Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models (ICML 2026 · similarity 0.89)
- FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.89)
- Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.