Research map / Vision-language and multimodal / Cross-modal learning

Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamic Mechanisms and Efficient Alignment

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds AdvBench-Omni (11 modality variants of AdvBench) to expose cross-modal safety gaps in omni-modal LLMs, analyzes refusal-vector magnitude shrinkage ("Mid-layer Dissolution"), and proposes OmniSteer, an SVD-derived golden refusal vector steered by lightweight layer-wise adapters.

Problem

Omni-modal LLMs have cross-modal safety vulnerabilities: refusal rates drop for combined-modality harmful inputs compared with single-modality inputs, and the internal mechanism behind this is not understood.

Approach

Decouple modality from semantics when building evaluation data, trace refusal-vector magnitude shrinkage across layers, extract a shared refusal direction via uncentered SVD over modality refusal vectors, and train per-layer adapters that predict adaptive steering strength along that direction.

Key results

• OmniSteer raises average RSR from 69.9% (vanilla) to 91.2%, versus 85.18% for Self-Reminder and 76.72% for OmniGuard, while keeping average BAR around 83.2%.
• Cross-modal refusal vectors have magnitude ratios of only 0.45-0.72 relative to the text refusal vector; magnitude shrinkage accounts for 88.3% of log-variance versus 9.8% for direction.
• On OmniBench, OmniSteer stays within about 2% of vanilla accuracy while Self-Reminder and OmniGuard lose about 8.1% on average, per the paper's claim.

Surprising findings

• Cross-modal combinations show a refusal-signal collapse in middle layers ('Mid-layer Dissolution') even though single-modality inputs are safe, with magnitude shrinkage rather than direction deviation driving the decline.
• Adapter-free steering with a fixed alpha trades off strongly: at alpha=0.1 RSR is about 99.8% but benign acceptance drops to about 46%.

Limitations

• Semantic separation for cross-modal prompts may confound results with reasoning difficulty; the appendix discussion acknowledges this concern.
• Evaluated only on three omni models and relies on LLM-as-judge (Qwen3-30B-A3B) for RSR and BAR.

Experimental setup

Datasets: AdvBench AdvBench-Omni AdvBench-MM HarmBench BeaverTails MM-SafetyBench HoliSafeBench VideoSafetyBench OmniSafetyBench OmniBench VLGuard AdvBench-Audio

Benchmarks: AdvBench-Omni HarmBench-Audio BeaverTails BeaverTails-Audio-1k MM-SafetyBench HoliSafeBench VideoSafetyBench OmniSafetyBench OmniBench

Metrics: Refusal Success Rate Benign Acceptance Rate accuracy Overall

Baselines: Self-Reminder OmniGuard

Builds on: refusal direction steering Self-Reminder OmniGuard AdvBench

Closest papers by meaning

• Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correction (NeurIPS 2026 · similarity 0.88)
• When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models (AAAI 2026 · similarity 0.87)
• Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.87)
• OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities (ICLR 2025 · similarity 0.87)
• Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.86)
• Refusal Direction is Universal Across Safety-Aligned Languages (NeurIPS 2025 · similarity 0.86)