Research map / Adversarial robustness and security / Attacks on text and graphs
Tug-of-War No More: Harmonizing Accuracy and Robustness in Vision-Language Models via Stability-Aware Task Vector Merging
Junhao Dong, Xinghua Qu, Cong Zhang, Sua Qi Rong, Nguyen Duc Thai, Wenbo Pan, Xinfeng Li, Tongliang Liu, Piotr Koniusz, Yew-Soon Ong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PISTOLE merging instead of PMG | PISTOLE · Zero-shot 14 datasets avg (eps 2/255) | clean+robust sum: 110.20 -> 116.89 | Improved |
| add GISM stability masks to vanilla merging (ablation) | PISTOLE · Zero-shot 14 datasets avg | clean+robust sum: 111.11 -> 113.47 | Improved |
| add PCI re-weighting on top of GISM (ablation) | PISTOLE · Zero-shot 14 datasets avg | clean+robust sum: 113.47 -> 114.83 | Improved |
| add adversarial parameter trajectory (APT) on top of GISM and PCI (ablation) | PISTOLE · Zero-shot 14 datasets avg | clean+robust sum: 114.83 -> 116.89 | Improved |
| use naturally fine-tuned VLM instead of pre-trained zero-shot VLM as natural task vector (ablation) | PISTOLE · Zero-shot 14 datasets avg | clean+robust sum: 114.93 -> 116.89 | Improved |
| use pre-trained VLM as natural vector: robust accuracy (ablation) | PISTOLE · Zero-shot 14 datasets avg | robust accuracy: 47.83 -> 47.65 | Hurt |
| use PISTOLE instead of PMG on ViT-H/14 | PISTOLE · Zero-shot 14 datasets, ViT-H/14 | clean+robust sum: 119.27 -> 126.06 | Improved |
| use PISTOLE instead of PMG at eps 4/255 | PISTOLE · Zero-shot avg, eps 4/255 | clean+robust sum: 92.69 -> 97.31 | Improved |
| use PISTOLE with LoRA instead of PMG with LoRA | PISTOLE · Zero-shot avg, eps 3/255 (LoRA) | clean+robust sum: 83.84 -> 88.82 | Improved |
| swap CLIP encoder for PISTOLE encoder in LLaVA 1.5 (COCO) | PISTOLE · COCO captioning, LLaVA 1.5 | CIDEr sum: 156.4 -> 165.5 | Improved |
| use PISTOLE instead of FARE for hallucination | PISTOLE · POPE hallucination | F1: 80.8 -> 83.0 | Improved |
| use PISTOLE merging weight lambda 0.2 (ablation) | PISTOLE · ImageNet adversaries | loss-parameter curvature | Improved |
Closest papers by meaning
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.90)
- SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models (ICML 2026 · similarity 0.90)
- Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices (ICML 2025 · similarity 0.89)
- Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models (ICML 2025 · similarity 0.89)
- R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning (CVPR 2025 · similarity 0.89)
- Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
