Research map / Vision-language and multimodal / Multimodal LLMs
Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes verbose images, imperceptible perturbations that make VLMs generate much longer captions, raising energy consumption and latency via delayed-EOS, uncertainty, and token-diversity losses plus temporal weight adjustment.
Problem
Energy-latency attacks on large vision-language models: adversarially inducing long generated sequences to inflate energy consumption and response time, threatening availability of VLM services.
Approach
Crafts image perturbations optimized with three losses (delayed EOS probability, token-level uncertainty via KL to uniform, sequence-level token diversity via nuclear norm of hidden states) combined with a temporal weight adjustment (log-decay weights plus momentum) under PGD, tailored to VLMs rather than LLMs or RNN captioners.
Key results
- Verbose images increase generated sequence length by 7.87x on MS-COCO and 8.56x on ImageNet relative to original images across four VLMs.
- On BLIP, verbose images reach 318.66 generated length on MS-COCO vs 179.42 for NICGSlowDown and 65.83 for sponge samples.
- Ablation on BLIP-2 shows the full three-loss combination with temporal decay and momentum gives the longest sequences (226.72 on MS-COCO vs 152.49 with none of the optimization modules).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use verbose images instead of original images | BLIP, BLIP-2, InstructBLIP, and MiniGPT-4 · MS-COCO and ImageNet | sequence length: 7.87× on MS-COCO; 8.56× on ImageNet | Helped |
| use verbose images instead of NICGSlowDown | BLIP, BLIP-2, InstructBLIP, and MiniGPT-4 · MS-COCO and ImageNet | sequence length | Helped |
| combine delayed EOS, uncertainty, and token diversity losses (ablation) | Verbose images on BLIP-2 · MS-COCO and ImageNet | sequence length: 177.95 -> 226.72 on MS-COCO; 217.78 -> 250.72 on ImageNet | Helped |
| add delayed EOS loss (ablation) | Verbose images on BLIP-2 · MS-COCO and ImageNet | sequence length: 139.54 -> 177.95 on MS-COCO; 161.46 -> 217.78 on ImageNet | Helped |
| add uncertainty loss (ablation) | Verbose images on BLIP-2 · MS-COCO and ImageNet | sequence length: 119.46 -> 177.95 on MS-COCO; 147.87 -> 217.78 on ImageNet | Helped |
| add token diversity loss (ablation) | Verbose images on BLIP-2 · MS-COCO and ImageNet | sequence length: 177.95 -> 226.72 on MS-COCO; 217.78 -> 250.72 on ImageNet | Helped |
| add temporal decay (ablation) | Verbose images on BLIP-2 without momentum · MS-COCO and ImageNet | sequence length: 152.49 -> 199.92 on MS-COCO; 144.90 -> 231.03 on ImageNet | Helped |
| add momentum (ablation) | Verbose images on BLIP-2 without temporal decay · MS-COCO and ImageNet | sequence length: 152.49 -> 187.32 on MS-COCO; 144.90 -> 214.92 on ImageNet | Helped |
| combine temporal decay and momentum (ablation) | Verbose images on BLIP-2 · MS-COCO and ImageNet | sequence length: 152.49 -> 226.72 on MS-COCO; 144.90 -> 250.72 on ImageNet | Helped |
| increase perturbation magnitude ε from 2 to 32 (ablation) | Verbose images on BLIP-2 · MS-COCO and ImageNet | sequence length: 91.75 -> 287.22 on MS-COCO; 103.51 -> 321.65 on ImageNet | Helped |
| increase perturbation magnitude ε from 2 to 32 (ablation) | Verbose images on BLIP-2 · MS-COCO and ImageNet | LIPIS: 0.0037 -> 0.1608 on MS-COCO; 0.0038 -> 0.1575 on ImageNet | Hurt |
| use verbose images instead of original images | BLIP-2 · VQA and visual reasoning | sequence length: 6.43 -> 271.95 on VQA; 5.70 -> 280.10 on visual reasoning | Helped |
Surprising findings
- Verbose images can induce hallucinated objects, with CHAIR_i rising from 11.41% to 79.93% on BLIP (MS-COCO).
- Instruction-tuned models sometimes respond with refusals such as "I am sorry, but I cannot describe the image", producing short outputs under the attack.
Limitations
- Attack length gains are smaller for some models, and length distributions are bimodal with some short outputs.
- Experiments are confined to a laboratory setting and not validated against real deployed services.
- Black-box transferability is weaker than the white-box setting.
Experimental setup
Datasets: MS-COCO ImageNet VQAv2 GQA
Benchmarks: MS-COCO ImageNet VQAv2 GQA
Metrics: generated sequence length energy consumption (J) latency (s) CHAIR LPIPS BLEU CIDEr
Baselines: Original images Random noise Sponge samples NICGSlowDown
Builds on: PGD adversarial attack NICGSlowDown sponge samples
Closest papers by meaning
- VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration (ICLR 2025 · similarity 0.90)
- Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.90)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.89)
- HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models (AAAI 2025 · similarity 0.89)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.89)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.