Research map / Vision-language and multimodal / Multimodal LLMs

Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CARE, a training-free VLM defense that uses causal mediation analysis to locate safety-critical mid-layer FFN channels and projects activations onto dual-modal (visual and textual) malicious subspaces via generalized eigen-decomposition, reducing jailbreak and PGD attack success rates.

Problem

LVLMs are vulnerable to jailbreak and adversarial attacks, and their internal safety mechanisms are opaque; existing activation-steering defenses localize unsafe components coarsely, ignore cross-modal interactions, and can degrade general capability.

Approach

Localize safety-critical layers and FFN modules via causal blocking and clustering analysis, attribute important visual and textual tokens with RBF-kernel dependence scores, derive malicious subspaces by generalized eigen-decomposition between benign and malicious activations, and at inference project FFN activations onto the complement with an adaptive visual-textual fusion.

Key results

• On LLaVA-OneVision, CARE reduces JailBreakV ASR from 45.71% (original) to 7.03%, versus 10.37% for the best baseline SPO-VLM.
• On Qwen2.5-VL, CARE reduces MM-SafetyBench ASR to 8.72% compared with 15.57% for ASTRA.
• General benchmark drops are small: about -1% on MMBench and -6% on MM-Vet for Qwen2.5-VL, with LLaVA-OneVision changes within ±3%.

Surprising findings

• Ablating FFN produces a much larger ASR increase than ablating MHSA, leading the authors to focus intervention on FFN activations.
• Over-rejection on OR-Bench-1k rises from 15.09% to 21.9% after CARE, a notable cost on hard benign inputs.

Limitations

• Over-rejection increases on OR-Bench-1k (15.09% to 21.9%), indicating a safety-utility trade-off.
• Several theoretical results (e.g., convergence corollary, optimal fusion theorem) are loosely argued and rely on informal steps.
• Evaluation relies on a single LLM judge (HarmBench-Llama2-13B) and a limited set of attacks and benchmarks.

Experimental setup

Datasets: JailBreakVBench AdvBench FigStep COCO 2017 Anthropic-HHH

Benchmarks: JailBreakVBench MM-SafetyBench PGD-Toxic PGD-Jailbreak APGD MI-FGSM MMBench MM-Vet ScienceQA OR-Bench-1k

Metrics: attack success rate accuracy over-rejection rate

Baselines: JailGuard Refusal Pairs CAST SPO-VLM ASTRA ShiftDC ProFS

Builds on: activation steering ASTRA SPO-VLM ShiftDC

Closest papers by meaning

• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.91)
• VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety (ICLR 2026 · similarity 0.91)
• Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility (ICML 2026 · similarity 0.90)
• Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models (ICLR 2026 · similarity 0.90)
• Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.90)
• Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs (ICLR 2026 · similarity 0.90)