Research map / Large language models / Instruction and fine-tuning
Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning
Yiwei Chen, Yuguang Yao, Yihua Zhang, Bingquan Shen, Gaowen Liu, Sijia Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Mixed-SFT safety fine-tuning with NPO unlearning | LLaVA-1.5-7B · VLGuard | ASR after one-word attack: 54.98% -> 12.92% | Improved |
| replace Mixed-SFT with RMU unlearning | LLaVA-1.5-7B · VLGuard | ASR after one-word attack: 54.98% -> 10.18% | Improved |
| replace Posthoc-SFT with RMU unlearning | LLaVA-1.5-7B · SPA-VL | ASR after one-word attack: 32.96% -> 22.64% | Improved |
| replace Mixed-SFT with NPO unlearning to reduce over-prudence | LLaVA-1.5-7B · VLGuard | refusal rate on benign queries: 91.76% -> 11.69% | Improved |
| replace Posthoc-SFT with RMU unlearning to reduce over-prudence | LLaVA-1.5-7B · SPA-VL | refusal rate on benign queries: 100.0% -> 19.24% | Improved |
| add Unsafe-Filter data filtering to unsafe base model | LLaVA-1.5-7B · VLGuard | ASR after one-word attack: 90.27% -> 90.72% | No clear effect |
| replace SFT with unlearning, utility cost | LLaVA-1.5-7B · VQAv2 | accuracy: 78.23% -> 77.04% | Mixed |
| use unlearning with 10% of the forget data (NPO) instead of full (ablation) | LLaVA-1.5-7B · VLGuard | ASR after one-word attack: 12.92% -> 18.77% | Hurt |
| use unlearning with 10% of the forget data (RMU) instead of full (ablation) | LLaVA-1.5-7B · VLGuard | ASR after one-word attack: 10.18% -> 22.12% | Hurt |
| add Gaussian noise to images under Mixed-SFT (ablation) | LLaVA-1.5-7B · VLGuard | ASR after one-word attack: 54.98% -> 54.08% | No clear effect |
| apply NPO unlearning rather than SFT under GCG attack | LLaVA-1.5-7B · VLGuard | ASR after GCG attack: 72.90% -> 3.87% | Improved |
Closest papers by meaning
- Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.92)
- Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs (ICLR 2026 · similarity 0.92)
- Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.92)
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.91)
- Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.91)
- SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
