Research map / Vision-language and multimodal / Multimodal LLMs

Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correction

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies Safety Geometry Collapse in multimodal LLMs, where modality-induced drift weakens refusal separability, and proposes ReGap, a training-free inference-time method that adaptively subtracts an estimated drift direction based on a self-rectification signal.

Problem

Multimodal LLMs fail to transfer text-only safety behavior to semantically equivalent audio, image, and omni-modal inputs, producing a multimodal safety gap that post-training defenses only partly close.

Approach

Decomposes multimodal safety into a text-aligned refusal direction and a modality-induced drift direction orthogonalized against refusal; subtracting the drift restores refusal separability, and a probe-based self-rectification score sets an input-dependent correction strength.

Key results

• On Qwen2.5-Omni-3B, ReGap reduces average Omni-SafetyBench ASR from 65.7% to 20.8% and MM-SafetyBench V-T ASR from 76.8% to 14.2%.
• On Qwen3-Omni-30B-A3B-Instruct, ReGap lowers average Omni-SafetyBench ASR from 54.5% to 27.5% while OmniBench overall accuracy rises slightly from 48.8 to 49.6.
• ShiftDC on MiniCPM-o-4.5 degrades MMAU accuracy from 76.2 to 29.2, whereas ReGap keeps it at 76.8.
• Automatic gpt-5-mini judge reaches 95.8% agreement with human consensus on 550 sampled outputs.

Surprising findings

• ShiftDC intervention causes severe MMAU collapse on Qwen2.5-Omni-3B (66.0 to 3.2) and MiniCPM-o-4.5 (76.2 to 29.2), indicating off-manifold decoding.
• Harmful inputs originally eliciting compliance show stronger self-rectification than those originally refused after drift correction.

Limitations

• Uses a single unified modality-drift direction rather than modality-specific drift subspaces.
• Depends on calibration data and model-specific hyperparameters (layers, threshold, coefficients).
• Only applicable to open-weight models whose hidden states can be intervened on.

Experimental setup

Datasets: Omni-SafetyBench AdvBench OmniInstruct MM-SafetyBench AudioJailBench OmniBench MMMU-Pro MMAU

Benchmarks: Omni-SafetyBench MM-SafetyBench AudioJailBench OmniBench MMMU-Pro MMAU

Metrics: attack success rate conditional attack success rate accuracy silhouette coefficient (CRS)

Baselines: VLGuard-SFT SPA-VL-DPO ShiftDC Refusal Steering

Builds on: representation engineering ShiftDC refusal direction difference-in-means steering

Closest papers by meaning

• When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models (AAAI 2026 · similarity 0.89)
• Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback (NeurIPS 2025 · similarity 0.89)
• Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.89)
• GuardAlign: Robust Safety Alignment in Multimodal Large Language Models (ICLR 2026 · similarity 0.89)
• Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory (CVPR 2026 · similarity 0.88)
• Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamic Mechanisms and Efficient Alignment (NeurIPS 2026 · similarity 0.88)