Research map / Vision-language and multimodal / Multimodal LLMs

Models as Lego Builders: Assembling Malice from Benign Blocks via Semantic Blueprints

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes StructAttack, a single-query black-box jailbreak that decomposes a harmful query into a topic plus benign-looking slot types, renders them as structured visual prompts (mind map, table, sunburst), and uses a completion-guided instruction to make LVLMs fill in harmful slot values.

Problem

LVLMs with safety alignment refuse direct harmful multimodal queries, and existing visual jailbreaks either rely on typography that OCR filters catch or on iterative optimization/OOD perturbations that need many queries.

Approach

Exploit semantic slot filling in reverse: encode a harmful query as a topic with locally benign slot types (plus distractor slots) rendered in structured diagrams, so the LVLM reconstructs the concealed intent and completes the missing slot values in one query.

Key results

• On Advbench-M, StructAttack (v2 table) reaches 69.0% ASR on GPT-4o versus 10.7% for FigStep-Pro and 10.2% for HADES.
• Average ASR is reported as 66.4% on closed-source LVLMs and 90.4% on open-source LVLMs (Qwen2.5VL-7B, InternVL-3-9B).
• Against a system-prompt defense on GPT-4o, StructAttack keeps 47.2% ASR while SI-Attack and JOOD drop to 2.9% and 4.3%.

Surprising findings

• Even text-only semantic slot filling achieved 54% ASR on GPT-4o, suggesting the vulnerability exists in text as well.
• Removing distractor slots and random layout perturbation reduced success only modestly, and the layout-agnostic ablation showed all structure types performed similarly.

Limitations

• Semantic drift on long or complex malicious queries can yield benign reconstructions.
• High-risk categories such as severe crimes sometimes trigger high-level educational refusals from GPT-4o.
• Human consistency check uses only 70 samples.

Experimental setup

Datasets: Advbench-M SafeBench

Benchmarks: Advbench-M SafeBench

Metrics: Attack Success Rate Harmfulness score (0-10)

Baselines: FigStep FigStep-Pro HADES SI-Attack JOOD

Builds on: FigStep HADES SI-Attack JOOD GCG

Closest papers by meaning

• Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.90)
• Jailbreaking Vision-Language Models Through the Visual Modality (ICML 2026 · similarity 0.90)
• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.88)
• Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility (ICML 2026 · similarity 0.88)
• JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models (NeurIPS 2025 · similarity 0.88)