Research map / Vision-language and multimodal / Multimodal LLMs

Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BadVision, a backdoor attack that compromises SSL vision encoders (CLIP ViT-L, EVA ViT-G) via adversarial trigger optimization and trigger-focusing backdoor learning, inducing attacker-chosen hallucinations in LLaVA-1.5 and MiniGPT-4 while evading DECREE detection.

Problem

Backdoor threats to SSL vision encoders shared and reused in large vision language models: existing SSL backdoors target classification and existing LVLM backdoors are text-specific, costly, or non-transferable.

Approach

Optimize a universal imperceptible adversarial trigger that pulls triggered features toward a target image's embedding, then fine-tune the encoder with a trigger-focusing loss that makes it ignore other adversarial perturbations, so feature concentration (the signal used by DECREE-style detectors) is not exposed.

Key results

• BadVision achieves 100% ASR on CLIP-based LLaVA-1.5 across five datasets while keeping benign performance (average drop about 1.4 points).
• Triggered outputs produce a 77.6% average relative visual understanding error across eight benchmarks on LLaVA-1.5.
• DECREE fails to flag BadVision (PL1 of 0.220 for CLIP and 0.498 for EVA, close to clean), whereas BadEncoder is detected (all below 0.1).

Measured effects reported in the paper

Surprising findings

• Universal adversarial attack and BadEncoder baselines behave differently: BadEncoder collapses benign performance even without the trigger, while standard adversarial perturbations barely affect LVLM outputs under the universal setting.

Limitations

• Triggers may be destroyed by image transformations when shared online, preventing activation.
• Bi-level optimization is solved only approximately with a two-stage local solution.
• Attack evaluated under standard input conditions only.

Experimental setup

Datasets: PASCAL VOC COCO Flickr30k VQAv2 GQA VizWiz POPE

Benchmarks: COCO Captions Flickr30k VizWiz Caption VQAv2 GQA POPE-adversarial POPE-popular POPE-random

Metrics: CIDEr VQA accuracy F1 attack success rate Sim-T Sim-B PL1 norm

Baselines: BadEncoder Adversarial attack (universal) Shadowcast ImgTroj Carlini et al. poisoning BadCLIP

Builds on: BadEncoder DECREE Shadowcast adversarial examples (PGD)

Closest papers by meaning

• Invisible Backdoor Attack against Self-supervised Learning (CVPR 2025 · similarity 0.89)
• Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift (CVPR 2025 · similarity 0.89)
• Attention-Imperceptible Backdoor Attacks on Vision Transformers (AAAI 2025 · similarity 0.89)
• DeDe: Detecting Backdoor Samples for SSL Encoders via Decoders (CVPR 2025 · similarity 0.89)
• BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimization (NeurIPS 2025 · similarity 0.87)
• Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment (CVPR 2024 · similarity 0.87)