Research map / Vision-language and multimodal / Multimodal LLMs
LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LlavaGuard, open VLM-based vision safeguards (0.5B–7B) trained on a human-annotated, rationale-augmented safety dataset with a customizable safety taxonomy and policy augmentation, outperforming VLMs and existing safeguards on balanced accuracy and policy exception score.
Problem
Existing image safety tools are rigid NSFW classifiers or closed VLM moderators that lack flexible, policy-adaptive, explainable safety assessments for vision datasets and generative models.
Approach
Fine-tune Llava-OneVision VLMs on a 5.5k-sample human-labeled safety dataset with guided LLM-generated rationales and policy-exception augmentation, so the model judges images against arbitrary natural-language safety policies and outputs JSON rating, category, and rationale.
Key results
- LlavaGuard-7B reaches 90.84% accuracy and 89.85 PES on the held-out test set, the best among all compared models including GPT-4o (72.92% accuracy, 77.29 PES).
- LlavaGuard-0.5B achieves 88.70% accuracy and 87.10 PES, outperforming all general VLMs and safeguards including 78B InternVL2.5.
- Human annotators agreed with LlavaGuard on 88% of safety ratings, 87% of categories and 81% of rationales.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune the VLM as LlavaGuard-7B rather than use Llava-OV-7B | Llava-OneVision-7B · Held-out test set | balanced accuracy: 60.81 -> 90.84 | Helped |
| fine-tune the VLM as LlavaGuard-0.5B rather than use Llava-OV-0.5B | Llava-OneVision-0.5B · Held-out test set | balanced accuracy: 52.00 -> 88.70 | Helped |
| use guided rather than base rationales | Llava-34B rationale generation · Guided rationale quality | mean score: 3.8 -> 9.1 | Helped |
| use Llava-34B rather than Llava-7B for guided rationale generation | Guided rationale generation · Rationale quality across model sizes | mean score: 7.0 -> 9.0 | Helped |
| evaluate only unambiguous ('Highly Unsafe' and 'Generally Safe') samples rather than the full test set (ablation) | LlavaGuard-7B · Unambiguous-only test set | balanced accuracy: 90.84% -> 93.91% | Helped |
| evaluate only ambiguous samples rather than the full test set (ablation) | LlavaGuard-7B · Ambiguous-only test set | balanced accuracy: 90.84% -> 87.77% | Hurt |
| use custom LlavaGuard policies instead of ImageGuard's default policy (ablation) | ImageGuard · Held-out test set | PES: 31.08 -> 27.00 | Hurt |
| remove approximately 20,000 images using LlavaGuard filtering | ResNet-50 trained on ImageNet · Filtered ImageNet, all classes | top-1 accuracy: 67.2 ± 0.5 -> 67.4 ± 0.4 | No clear effect |
| remove approximately 20,000 images using LlavaGuard filtering | ResNet-50 trained on ImageNet · Filtered ImageNet, all classes | top-5 accuracy: 87.2 ± 0.6 -> 87.1 ± 0.6 | No clear effect |
| remove safety-filtered images | ResNet-50 trained on the ten most heavily filtered ImageNet classes · Ten most heavily filtered ImageNet classes | top-1 accuracy: 53.6 ± 3.5 -> 46.9 ± 4.8 | Hurt |
| remove safety-filtered images | ResNet-50 trained on the ten most heavily filtered ImageNet classes · Ten most heavily filtered ImageNet classes | top-5 accuracy: 82.4 ± 3.1 -> 77.3 ± 3.6 | Hurt |
Surprising findings
- LlamaGuard-3-11B-Vision has almost negligible recall (0.56%) despite high precision, misclassifying nearly all unsafe images as safe.
- ImageGuard shows nearly identical results under its default policy and custom policies, suggesting it ignores the supplied policy.
Limitations
- Human supervision covered only category and rating; rationales were generated synthetically by Llava-34B.
- Annotation was largely guided by the default policy, so policy-varied labels are limited.
- Errors persist on borderline cases and images with complex embedded text; the test set has a ceiling near 94% due to labeling inconsistency.
Experimental setup
Datasets: SMID web-crawled images from Google and Bing ImageNet CC12M COCO I2P Stylebreeder
Benchmarks: LlavaGuard held-out test set (824 samples) ImageNet dataset audit I2P generated images audit
Metrics: balanced accuracy recall precision policy exception score (PES) policy exception rate (PER)
Baselines: Llava-OV-0.5B Llava-OV-7B InternVL2.5 Qwen2.5-VL QVQ-72B-Preview GPT-4o LlamaGuard-3-11B-Vision OpenAI omni-moderation ImageGuard Siglip2Guard NSFW classifiers Q16
Builds on: LLaVA-OneVision Qwen2.5-VL SMID
Closest papers by meaning
- Towards Policy-Adaptive Image Guardrail: Benchmark and Method (CVPR 2026 · similarity 0.90)
- Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models (ICLR 2026 · similarity 0.88)
- SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.88)
- VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety (ICLR 2026 · similarity 0.87)
- ELITE: Enhanced Language-Image Toxicity Evaluation for Safety (ICML 2025 · similarity 0.86)
- How Does Vision-Language Adaptation Impact the Safety of Vision Language Models? (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.