Research map / Vision-language and multimodal / Multimodal LLMs

SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes where, how, and which harmful multimodal tokens trigger jailbreaks in MLLMs, then proposes SafePTR, a training-free defense that prunes top-K semantically deviant visual/text tokens in vulnerable early-middle layers and restores benign features in later layers.

Problem

Multimodal jailbreak attacks bypass safeguards in MLLMs; existing defenses (image-to-text translation, safe prompting, multimodal safety tuning) lack understanding of which harmful tokens trigger jailbreaks, leading to text-driven vulnerability, overdefense, and high training cost.

Approach

Identify vulnerable early-middle layers via layer-wise intervention and find that under 1% of tokens deviate strongly from a safety-aligned instruction's hidden states; prune the top-K most deviant tokens in those layers (HTP) and restore benign features in subsequent layers (BFR), with no training.

Key results

• On JailbreakV-28K, SafePTR reaches 0.98 average ASR on LLaVA-1.5 (vs. 2.10 for Immune and 14.36 for AdaShield) and 2.46 on DeepSeek-VL2.
• On FigStep, SafePTR reduces average ASR on LLaVA-1.5-7B to 1.60 (vs. 4.23 for Immune and 13.00 for AdaShield).
• SafePTR attains 1538.1 on MME for LLaVA-1.5-7B, above the 1503.6 original baseline, with no training data and ~3.67 s latency.

Measured effects reported in the paper

Surprising findings

• Less than 1% of multimodal tokens in early-middle layers cause significant semantic deviation and drive jailbreaks.
• Pruning harmful tokens without any safety tuning substantially lowers ASR, while restoring benign features in later layers preserves utility.

Limitations

• Relies on intermediate hidden states to compute semantic deviation, so it is not applicable to black-box models such as GPT-4.
• Fixed Top-K pruning may not adapt to inputs with varying risk levels.
• Layer windows and K were chosen empirically per model.

Experimental setup

Datasets: FigStep MM-SafetyBench JailbreakV-28K

Benchmarks: JailbreakV-28K MM-SafetyBench FigStep MME MM-Vet

Metrics: attack success rate inference latency training data size

Baselines: Immune AdaShield ECSO CoCA FigStep

Builds on: AdaShield Immune FigStep ECSO

Closest papers by meaning

• Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy (CVPR 2025 · similarity 0.93)
• Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment (CVPR 2025 · similarity 0.93)
• Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models (NeurIPS 2025 · similarity 0.91)
• Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models (ICLR 2025 · similarity 0.91)
• MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMs (ICLR 2026 · similarity 0.91)
• The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense (NeurIPS 2025 · similarity 0.90)