Research map / Adversarial robustness and security / Attacks on text and graphs

Forced Deferral: Manipulating Routing Decisions in Multimodal LLM Cascades

NeurIPS 2026 · Adversarial robustness and security › Attacks on text and graphs · approach: Adversarial prompts for language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the Forced Deferral Attack, a universal border image trigger trained with a temperature-flattened KL objective that lowers weak MLLM confidence and forces cascades to route queries to the strong model.

Problem

Confidence-based MLLM cascades defer uncertain queries to a strong model; an adversary can manipulate the weak model's confidence to force unnecessary deferrals, wasting compute and gaining higher-quality answers without targeting correctness.

Approach

Learn a single border-only image trigger that matches the weak model's next-token distributions on triggered inputs to temperature-flattened versions of its clean logits (teacher-forced along clean responses), reducing confidence in a metric-agnostic way while preserving central visual content.

Key results

• On Gemma same-family cascade under AP on MMBench, the FDA trigger raises reroute rate close to 1 at low calibrated deferral rates, with system accuracy reaching about 90% on MMBench (Ablation: 0.8974 at D=0.1 vs clean 0.7436).
• Cross-family FDA (Gemma4-E2B weak routed to Qwen3-VL-32B or similar strong model) matches same-family FDA within about 0.04 accuracy on most settings (e.g., ScienceQA Gemma 0.956 vs 0.873 same-family at D=0.1 in Table).
• FDA lowers mean AP confidence from 0.94 to 0.03 on Qwen3-VL-2B for MMBench, and remains effective under JPEG, Gaussian noise, and bit-depth defenses.

Surprising findings

• Attacked inputs routed to the strong model still yield higher system accuracy, meaning the attack gains quality for the adversary at the provider's compute cost.
• JPEG compression is the strongest preprocessing defense but still does not restore clean-cascade routing behavior.

Limitations

• Evaluated only on image-based multiple-choice tasks with short answers, not open-ended or multi-turn settings.
• Only three confidence metrics and two model families tested; attacker assumes access to weak-model logits or a same-family surrogate.
• Border trigger occupies visual space, and larger widths reduce strong-model accuracy.

Experimental setup

Datasets: MMBench MMMU ScienceQA

Benchmarks: MMBench MMMU ScienceQA

Metrics: reroute rate system accuracy weak model confidence (AP, 1/PPL, Jaccard)

Baselines: defocus blur prompt injection clean cascade max-entropy border trigger global adversarial noise

Builds on: confidence-based LLM cascades temperature scaling KL distillation objective

Closest papers by meaning

• Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models (NeurIPS 2024 · similarity 0.88)
• REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective (ICML 2025 · similarity 0.87)
• Scaling Laws for Adversarial Attacks on Language Model Activations and Tokens (ICLR 2025 · similarity 0.86)
• On the Robustness of Large Multimodal Models Against Image Adversarial Attacks (CVPR 2024 · similarity 0.86)
• RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content (ICML 2024 · similarity 0.86)
• Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (CVPR 2025 · similarity 0.86)