Research map / Vision-language and multimodal / Human and embodied modeling
Hyperbolic Safety-Aware Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HySAC, a hyperbolic CLIP fine-tuned on ViSU with contrastive and entailment losses that place safe and unsafe image-text pairs in an entailment hierarchy, enabling NSFW classification and safety-aware retrieval with traversal.
Problem
Vision-language models like CLIP retrieve unsafe content; existing unlearning approaches erase unsafe knowledge and thus reduce the model's ability to distinguish safe from unsafe content.
Approach
Shift from unlearning to awareness by encoding safe and unsafe image-text pairs as an entailment hierarchy in Lorentz hyperbolic space, with safe content near the origin and unsafe content farther away, plus a traversal mechanism that moves query embeddings toward safe or unsafe regions.
Key results
- On unsafe-to-safe text-to-image retrieval (T*-to-I ∪ I*), HySAC reaches R@1 30.5 vs 8.0 for Safe-CLIP and 2.0 for CLIP.
- On NudeNet NSFW classification, HySAC achieves 99.5% accuracy versus 97.3% for the CLIP-classifier baseline.
- On real NSFW retrieval (SMID, image-to-text), HySAC retrieves 97.9% safe items versus 59.1% for CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HySAC instead of Safe-CLIP for unsafe-query-to-safe retrieval | CLIP ViT-L/14 · ViSU test set, T⋆-to-I∪I⋆ | R@1: 8.0 -> 30.5 | Helped |
| Use HySAC instead of Safe-CLIP for unsafe-query-to-safe retrieval | CLIP ViT-L/14 · ViSU test set, I⋆-to-T∪T⋆ | R@1: 19.1 -> 42.1 | Helped |
| Use HySAC instead of CLIP | CLIP ViT-L/14 · ViSU test set, unsafe-only T⋆-to-I⋆ retrieval | R@1: 73.1 -> 81.4 | Helped |
| Use HySAC instead of CLIP | CLIP ViT-L/14 · ViSU test set, unsafe-only I⋆-to-T⋆ retrieval | R@1: 72.8 -> 82.2 | Helped |
| Remove modality-entailment losses (ablation) | HySAC · ViSU test set, T-to-I safe retrieval | R@1: 49.8 -> 52.3 | Helped |
| Remove modality-entailment losses (ablation) | HySAC · ViSU test set, I-to-T safe retrieval | R@1: 48.2 -> 50.8 | Helped |
| Add modality-entailment losses (ablation) | HySAC · ViSU test set, T⋆-to-I∪I⋆ safe retrieval | R@1: 4.1 -> 30.5 | Helped |
| Add modality-entailment losses (ablation) | HySAC · ViSU test set, I⋆-to-T∪T⋆ safe retrieval | R@1: 5.5 -> 42.1 | Helped |
| Remove safety-entailment loss (ablation) | HySAC · ViSU test set, T-to-I safe retrieval | R@1: 49.8 -> 51.0 | Helped |
| Remove safety-entailment loss (ablation) | HySAC · ViSU test set, I-to-T safe retrieval | R@1: 48.2 -> 49.8 | Helped |
| Add safety-entailment loss (ablation) | HySAC · ViSU test set, T⋆-to-I∪I⋆ safe retrieval | R@1: 1.4 -> 30.5 | Helped |
| Add safety-entailment loss (ablation) | HySAC · ViSU test set, I⋆-to-T∪T⋆ safe retrieval | R@1: 7.4 -> 42.1 | Helped |
Surprising findings
- Removing the entailment losses slightly improves safe-only retrieval but substantially hurts unsafe-content retrieval.
- On Mixed NSFW, HySAC has a higher false positive rate (16.5%) than several baselines, despite its higher recall on unsafe content.
Limitations
- Real unsafe data with correlated safe alternatives was unavailable, so that retrieval aspect could not be evaluated.
- Traversal into safe regions may reduce relevance of retrieved content.
- Higher false positive rate on Mixed NSFW classification than some baselines.
Experimental setup
Datasets: ViSU NudeNet NSFW data source URLs SMID LAION-400M Mixed NSFW PASS
Benchmarks: ViSU test set NudeNet NSFW data source URLs SMID Mixed NSFW
Metrics: Recall@1 Recall@10 Recall@20 accuracy false positive rate false negative rate % safe retrieved
Baselines: CLIP Safe-CLIP MERU HyCoCLIP NSFW-CNN CLIP-classifier CLIP-distance NudeNet Q16
Builds on: CLIP MERU HyCoCLIP Safe-CLIP hyperbolic entailment cones
Closest papers by meaning
- VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety (ICLR 2026 · similarity 0.90)
- Compositional Entailment Learning for Hyperbolic Vision-Language Models (ICLR 2025 · similarity 0.89)
- Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility (ICML 2026 · similarity 0.88)
- Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models (ICLR 2026 · similarity 0.88)
- Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection (CVPR 2026 · similarity 0.87)
- Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.