Research map / Vision-language and multimodal / Multimodal LLMs

Steal the Patch Size: Adversarially Manipulate Vision Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that a grid-size counting probe exhibits periodic accuracy collapses in black-box VLMs (Patch-Size Matching), and uses a three-stage black-box procedure with padding and consistency tests to recover patch size and resize resolution, then demonstrates model-targeted adversarial transfer attacks.

Problem

Private vision-tokenizer configurations of deployed VLMs (visual patch size, dynamic vs fixed resizing, target resolution) are undisclosed and treated as secret; the paper asks whether they can be recovered through standard black-box API queries alone.

Approach

ViT patchification creates a task-level side channel: when synthetic grid cell sizes align with the hidden patch grid, boundary cues vanish, producing periodic accuracy valleys. Sweeping cell size, adding zero-padding to rescale the effective grid, and using hypothesis tests and paired consistency checks reveals P and S, which then enable preprocessing-aware model-targeted adversarial attacks.

Key results

• Qwen3-VL 30B shows stable accuracy minima at D in {48,64,80}, implying patch size P=16; Claude 3.7 minima at multiples of 14, implying P=14.
• For GPT-4o, the consistency test selects target resolution S=512 (z>=10 vs 448, 480, 544), implying P=16.
• Targeted GPT-4o attack raises GPT-4o ASR to 86.5% from 78.5% baseline while lowering Claude 3.7 ASR to 4.5% from 13.5%.

Surprising findings

• Claude 4.5, despite stronger reasoning, shows the same patch-aligned blind spot as Claude 3.7.
• GPT-4o's unpadded sweep is smooth, appearing to hide periodicity, while padding reveals it at period scaling with canvas size.

Limitations

• Relies on fixed-resolution or dynamic preprocessing without content-adaptive cropping; non-identifiable otherwise.
• Model-targeted attack benefit depends on differing preprocessing pipelines between target and non-target models.
• Results for proprietary models rely on inferred configurations that cannot be verified directly; GPT-4o detail modes add complexity.

Experimental setup

Datasets: synthetic grid images NIPS 2017 Adversarial Challenge

Benchmarks: synthetic grid-size counting probe NIPS 2017 Adversarial Challenge (1,000 images)

Metrics: accuracy vs cell size Welch t-statistic permutation p-value z-statistic consistency win counts attack success rate

Baselines: SSA-CWA AnyAttack Baseline transfer attack (model-agnostic)

Builds on: Vision Transformer patchification SSA-CWA AnyAttack

Closest papers by meaning

• Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.90)
• Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks (CVPR 2025 · similarity 0.87)
• VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models (ICML 2026 · similarity 0.87)
• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.87)
• Failures to Find Transferable Image Jailbreaks Between Vision-Language Models (ICLR 2025 · similarity 0.87)
• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (ICLR 2024 · similarity 0.87)