Research map / Vision-language and multimodal / Multimodal LLMs

TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TreeTeaming, an LLM-orchestrated hierarchical strategy-tree framework that autonomously discovers and refines multimodal jailbreak strategies against VLMs, reporting SOTA ASR on 11 of 12 target models.

Problem

Existing VLM red-teaming and jailbreak methods rely on predefined, manually designed strategies, which limits discovery of novel and diverse vulnerabilities.

Approach

An LLM Orchestrator grows a hierarchical Strategy Tree (parent categories with leaf strategies) from one seed example, deciding each round whether to exploit high-ASR leaf strategies or explore new ones; a tool-equipped multimodal Actuator instantiates strategies and a consistency checker filters off-target samples.

Key results

• Achieves 87.60% ASR on GPT-4o and 61.60% on Claude-3.5, with SOTA on 11 of 12 VLMs and average ASR of 89.48%.
• Visual diversity 0.641 vs 0.500 for Trust-VLM; textual toxicity 6.63% and visual toxicity 9.86%, lowest among compared methods.
• Strategy tree yields kNN-distance 0.275-0.282 vs union-set 0.263 at n=15, i.e., about 7.2% higher on the 15-strategy setting.

Surprising findings

• Upgrading the Actuator LLM (to Gemini-2.5-Pro) gives larger ASR gains than upgrading the Orchestrator, e.g. +13.4% vs +4.2% on Qwen3-VL-235B.
• Injecting a salient decoy object into FigStep images raises GPT-4o ASR from 3.40% to 87.50%.

Limitations

• Baselines Arondight and Trust-VLM lack released code, so results for many models are unavailable and copied from papers with different iteration budgets.
• Strategy consistency checker relies on a 7B VLM and was validated on only 200 samples.

Experimental setup

Datasets: SafeBench (500 malicious questions) MM-SafetyBench

Benchmarks: SafeBench MM-SafetyBench

Metrics: attack success rate textual diversity (1-Self-BLEU) visual diversity (CLIP cosine) kNN-distance kNN-entropy toxicity (unsafe proportion)

Baselines: FigStep MML MM-SafetyBench Hades CS-DJ JOOD SI-Attack Arondight Trust-VLM Vanilla prompting

Builds on: Tree of Attack (TAP) AutoDAN-Turbo AutoRedTeamer STAR

Closest papers by meaning

• ReaLM: A Unified Red-Teaming Benchmark for Physical-World VLMs (NeurIPS 2026 · similarity 0.90)
• Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.89)
• There are Levels to It: Red Teaming LLMs with Hierarchical Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
• DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints (AAAI 2025 · similarity 0.89)
• RedVLA: Physical Red Teaming for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
• Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning (ICLR 2025 · similarity 0.89)