Research map / Adversarial robustness and security / Attacks on text and graphs
AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization
Chaohu Liu, Gui Tianyi, Yu Liu, Linli Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AdPO-trained CLIP encoder instead of FARE or TeCoA encoders in LVLMs | LLaVA 1.5-7B · COCO captioning l_inf 2/255 | CIDEr: FARE2 53.6 -> AdPO2 62.7 adversarial; clean 109.9 -> 114.3 | Improved |
| train robust encoder on small TinyLLaVA (OpenELM-450M) and transfer to OpenFlamingo-9B and LLaVA | AdPO · OpenFlamingo-9B COCO | CIDEr: OF-9B COCO clean 79.6, 2/255 38.2 vs FARE2 34.2 | Improved |
| increase training perturbation radius from 2/255 to 4/255 (ablation) | AdPO · LLaVA 1.5-7B COCO | CIDEr: clean 114.3 -> 109.2; l_inf 4/255 40.7 -> 47.8 | Mixed |
| swap DPO for SimPO (no reference model) (ablation) | AdPO · VQAv2 | VQA accuracy clean / adversarial: SimPO 68.1 / 30.7; IPO 69.2 / 32.6 | Hurt |
| swap DPO for IPO, KTO or StepDPO (ablation) | AdPO · VQAv2 | VQA accuracy: clean 69.2-70.1; adversarial 32.0-32.6 | No clear effect |
| evaluate against C&W or sC&W attacks instead of APGD (ablation) | AdPO · VQAv2 | adversarial accuracy: APGD 32.4; C&W 33.1; sC&W 32.8 | No clear effect |
| increase lambda weight on adversarial image optimization (ablation) | AdPO · VQAv2 | clean vs adversarial accuracy | Mixed |
| use AdPO encoder against targeted attacks | LLaVA 1.5-7B · Targeted attack, eps=4/255 | mean ASR: 100% (CLIP) -> 0% (AdPO2) | Improved |
Closest papers by meaning
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.90)
- R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning (CVPR 2025 · similarity 0.89)
- Revisiting Adversarial Training at Scale (CVPR 2024 · similarity 0.89)
- Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.88)
- Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
