Research map / Adversarial robustness and security / Attacks on text and graphs
Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASTRA, a training-free VLM jailbreak defense that builds steering vectors from Lasso-attributed visual tokens of PGD adversarial images and applies calibrated, projection-based adaptive activation steering at inference.
Problem
VLMs are vulnerable to perturbation-based and structured (typography) jailbreak attacks, while existing defenses require costly purification, adversarial training, or multiple inference passes for response evaluation.
Approach
Attribute jailbreak-relevant visual tokens via random ablation and a Lasso surrogate, use their activation difference as a harmful steering vector, and steer activations only when they project positively onto that vector, with activations calibrated by a mean activation to avoid similar-angle clustering.
Key results
- On MiniGPT-4, ASTRA reduces toxicity score to 4.51% at eps=64/255 vs 54.70% undefended and 16.51% for JailGuard at eps=16/255 (11.30% vs 16.51%).
- On Qwen2-VL, ASTRA achieves ASR of 6.06% at eps=16/255 vs 67.27% undefended.
- ASTRA inference time per token matches the undefended model (e.g., 173.77 ms vs 173.19 ms on MiniGPT-4) while JailGuard is about 9x slower (1557.98 ms).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ASTRA adaptive activation steering instead of the strongest listed baseline | MiniGPT-4-13B · Perturbation-based attacks, ε=16/255 to unconstrained | Toxicity Score (%): 16.51 -> 11.30; 18.93 -> 8.84; 20.93 -> 4.51; 21.23 -> 4.48 | Helped |
| Use ASTRA adaptive activation steering instead of the strongest listed baseline | MiniGPT-4-13B · Perturbation-based jailbreak attacks, ε=16/255 to unconstrained | ASR (%): 20.00 -> 9.09; 22.73 -> 13.18; 17.27 -> 15.46; 16.36 -> 9.09 | Helped |
| Use ASTRA adaptive activation steering instead of the strongest listed baseline | Qwen2-VL-7B · Perturbation-based attacks, ε=16/255 to unconstrained | Toxicity Score (%): 29.37 -> 15.52; 24.68 -> 5.45; 28.74 -> 2.39; 27.76 -> 0.07 | Helped |
| Use ASTRA adaptive activation steering instead of the strongest listed baseline | Qwen2-VL-7B · Perturbation-based jailbreak attacks, ε=16/255 to unconstrained | ASR (%): 19.09 -> 6.06; 20.00 -> 5.00; 21.82 -> 18.18 | Helped |
| Add calibration activation to the projection term (ablation) | ASTRA on Qwen2-VL-7B · Perturbation-based attacks, ε=16/255 to unconstrained | Toxicity Score (%): 40.49 -> 15.52; 41.80 -> 5.45; 33.90 -> 2.39; 10.50 -> 0.07 | Helped |
| Add calibration activation to the projection term (ablation) | ASTRA on Qwen2-VL-7B · Perturbation-based attacks, ε=16/255 to unconstrained | ASR (%): 50.00 -> 6.06; 24.55 -> 5.00; 51.82 -> 18.18; 72.73 -> 15.45 | Helped |
| Use image-attributed visual tokens instead of entire-image activations (ablation) | ASTRA on Qwen2-VL-7B with calibration · Perturbation-based attacks, ε=16/255 to unconstrained | Toxicity Score (%): 37.70 -> 15.52; 35.28 -> 5.45; 21.59 -> 2.39; 5.24 -> 0.07 | Helped |
| Use image-attributed visual tokens instead of entire-image activations (ablation) | ASTRA on Qwen2-VL-7B with calibration · Perturbation-based attacks, ε=16/255 to unconstrained | ASR (%): 46.82 -> 6.06; 47.28 -> 5.00; 42.73 -> 18.18; 22.73 -> 15.45 | Helped |
| Steer with random Gaussian noise instead of an image-derived steering vector (ablation) | ASTRA on Qwen2-VL-7B · Perturbation-based attacks, ε=16/255 to unconstrained | Toxicity Score (%): 44.10; 53.80; 61.09; 55.10 | Hurt |
| Steer with random Gaussian noise instead of an image-derived steering vector (ablation) | ASTRA on Qwen2-VL-7B · Perturbation-based attacks, ε=16/255 to unconstrained | ASR (%): 64.55; 67.27; 69.09; 76.36 | Hurt |
Surprising findings
- Steering with textual jailbreak templates fails to counteract perturbation-based image attacks, suggesting textual and visual jailbreaks use different mechanisms.
- Steering vectors built from unconstrained-epsilon adversarial images do not transfer well to bounded-epsilon attacks in several settings.
Limitations
- Steering vectors from unconstrained adversarial images transfer imperfectly to bounded-epsilon attacks
- Adaptive attack results remain nontrivial for some settings (e.g., Qwen2-VL ASR 58-69% under adaptive attack)
- Hyperparameters (alpha, k, layer) are tuned on validation sets
Experimental setup
Datasets: ImageNet AdvBench Anthropic-HHH RealToxicityPrompts MM-SafetyBench JailbreakBench MM-Vet MMBench XSTest
Benchmarks: Detoxify toxicity score on RealToxicityPrompts with PGD images HarmBench ASR on AdvBench and Anthropic-HHH MM-Vet MMBench XSTest MM-SafetyBench JailbreakBench
Metrics: toxicity score attack success rate utility score perplexity inference time per token
Baselines: Self-reminder JailGuard ECSO Refusal Pairs steering Jailbreak Templates steering
Builds on: activation steering representation engineering PGD adversarial attack JailGuard ECSO
Closest papers by meaning
- JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models (CVPR 2025 · similarity 0.90)
- Failures to Find Transferable Image Jailbreaks Between Vision-Language Models (ICLR 2025 · similarity 0.90)
- Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.90)
- Retention Score: Quantifying Jailbreak Risks for Vision Language Models (AAAI 2025 · similarity 0.90)
- MMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Vision Language Models (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.