Research map / Adversarial robustness and security / Attacks on text and graphs
Test-Time Attention Purification for Backdoored Large Vision Language Models
Zhifang Zhang, Bojun Yang, Shuo He, Weitong Chen, Wei Emma Zhang, Olaf Maennel, Lei Feng, Miao Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CleanSight attention-based detection and pruning instead of pixel-space purification | backdoored LLaVA-1.5-7B · VQAv2 (BadNet, adapter) | ASR (%): 100.00 -> 0 | Improved |
| use CleanSight instead of ZIP across attacks | backdoored LLaVA-1.5-7B · VQAv2 (Blended) | ASR (%): 84.77 -> 0 | Improved |
| add CleanSight on clean inputs | backdoored LLaVA-1.5-7B · VQAv2 (BadNet), clean utility | V-score: 62.89 -> 62.63 | No clear effect |
| add attention-stealing detection before pruning (instead of pruning all inputs) (ablation) | CleanSight · VQAv2 (BadNet), clean utility | V-score: 52.47 -> 62.63 | Improved |
| use attention-guided selective pruning instead of random 90% visual token removal (ablation) | CleanSight · VQAv2 (BadNet) | ASR (%): 100.00 -> 0 | Improved |
| apply pixel perturbation (transformation defense) to backdoored LVLM | LLaVA backdoored model · Figure 1 attack success rate | ASR | No clear effect |
| flatten visual attention weights toward uniform | backdoored LLaVA · Figure 1 attack success rate | ASR | Improved |
| lower pruning threshold tau to about 1e-3 (oracle detector) (ablation) | CleanSight · VQAv2 (four attacks) | ASR (%): ASR drops to nearly zero | Improved |
| make tau overly small (ablation) | CleanSight · VQAv2 poisoned utility | V-score on poisoned input | Hurt |
| increase clean validation set size (ablation) | CleanSight detector · BadNet and Blended detection | FPR | Improved |
| use head-specific attention ratio instead of head-averaged ratio (ablation) | CleanSight detector · AUROC across layers | AUROC | Improved |
| apply CleanSight to other LVLMs and a LoRA-tuned model | LLaVA-1.5-7B LoRA, LLaVA-1.5-13B, InstructBLIP-7B, Qwen-VL2-7B · VQAv2 (BadNet) | ASR (%): LoRA 100.00 -> 5.08; Qwen-VL2 100.00 -> 0 | Improved |
Closest papers by meaning
- Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift (CVPR 2025 · similarity 0.91)
- Backdoor Cleaning without External Guidance in MLLM Fine-tuning (NeurIPS 2025 · similarity 0.90)
- Attention-Imperceptible Backdoor Attacks on Vision Transformers (AAAI 2025 · similarity 0.88)
- SEER: Backdoor Detection for Vision-Language Models through Searching Target Text and Image Trigger Jointly (AAAI 2024 · similarity 0.88)
- Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models (CVPR 2025 · similarity 0.88)
- Backdooring Vision-Language Models with Out-Of-Distribution Data (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
