Research map / Vision-language and multimodal / Multimodal LLMs

Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models

ICML 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that VLLM instruction fine-tuning erodes LLM safety and that harmful data in VL instruction sets contributes; releases VLGuard, a 2,000-image VL safety instruction dataset, and shows post-hoc or mixed fine-tuning with it greatly reduces attack success while keeping helpfulness.

Problem

Vision large language models are prone to generating harmful content and to simple jailbreaks, partly because vision-language instruction fine-tuning erodes the safety alignment inherited from the underlying LLM and training data contains harmful samples.

Approach

Curate VLGuard, a vision-language safety instruction-following dataset generated with GPT-4V covering harmful image and instruction categories, and use it either mixed into standard VLLM training or as a cheap post-hoc fine-tuning step paired with a small amount of helpfulness data.

Key results

• Post-hoc fine-tuning of LLaVA-v1.5-7B with VLGuard plus 5,000 LLaVA samples reduces VLGuard Safe-Unsafe ASR from 87.46% to 0.90% and Unsafe ASR from 72.62% to 0.23%, with helpfulness maintained.
• Mixed fine-tuning with VLGuard (about 0.3% of LLaVA stage-2 data) reduces FigStep ASR for LLaVA-v1.5-7B from 90.40% to 0.00% while raising the VLGuard Safe-Safe score from 18.82 to 20.78.
• Black-box TAP attack success on LLaVA-v1.5-7B drops from 62% to 20% (mixed) and 34% (post-hoc).

Measured effects reported in the paper

Surprising findings

• Fine-tuning on safety data alone causes exaggerated safety (XSTest safe answer rate drops to 41.6%), so helpfulness data must be mixed in.
• LoRA fine-tuning is less safe than full fine-tuning on identical data, and removing harmful LLaVA samples only partially restores safety.

Limitations

• Defense is not robust to sophisticated attacks; white-box adversarial attacks raise toxicity of fine-tuned models, though still below the original.
• Training set limited to 2,000 images due to GPT-4V cost.
• Llama-Guard has false negatives, so string matching is the primary metric.

Experimental setup

Datasets: VLGuard train LLaVA-v1.5 instruction data Alpaca Privacy Alert Hateful Memes Harmful Political Memes Harmful Object Dataset Bad Ads AdvBench XSTest FigStep MMLU

Benchmarks: AdvBench XSTest FigStep VLGuard MMLU AlpacaEval 2.0 ScienceQA VizWiz MMBench MM-Vet

Metrics: attack success rate win rate accuracy human win rate

Baselines: LLaVA-v1.5-7B LLaVA-v1.5-13B MiniGPT-v2 Safety LLaMA InstructBLIP Qwen-VL-Chat CogVLM InternLM-XComposer

Builds on: LLaVA-v1.5 MiniGPT-v2 FigStep Llama-Guard

Closest papers by meaning

• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.93)
• How Does Vision-Language Adaptation Impact the Safety of Vision Language Models? (ICLR 2025 · similarity 0.92)
• Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models (ICLR 2025 · similarity 0.92)
• Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.92)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.91)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.91)