Research map / Vision-language and multimodal / Human and embodied modeling

PoisonedEye: Knowledge Poisoning Attack on Retrieval-Augmented Generation based Large Vision-Language Models

ICML 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Multimodal retrieval-augmented modeling

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PoisonedEye, a knowledge poisoning attack on vision-language RAG systems that injects a single image-text poison sample optimized to be retrieved for a target query or an entire class of queries and to induce a target response.

Problem

Vision-language RAG systems that retrieve from multimodal knowledge databases are vulnerable to poisoning, but prior RAG poisoning work only targeted text. The paper asks whether a single injected image-text pair can make an LVLM output an attacker-chosen response to a target query or query class.

Approach

Extends PoisonedRAG to multimodal settings: an LLM crafts a misleading poison text for inducibility, and signed gradient descent on a bounded perturbation of the poison image minimizes retrieval distance to the target query (single query) or to the estimated center of an image class (class query) for retrievability.

Key results

• PoisonedEye-C achieves an average RSR-1 of 86.91%, RSR-K of 92.41% and ARD of 0.7906 across retrievers and datasets, versus 49.69% RSR-1 for the baseline PoisonedEye-B.
• PSR closely tracks RSR-K across settings, e.g. PoisonedEye-C reaches 92.63% PSR on LLaVA-v1.6-Mistral-7B with SigLIP-so400m on Places-365.
• Under RoCLIP defense, the original attack's PSR drops to 0%, but the enhanced attack retains 54.52% PSR.

Measured effects reported in the paper

Surprising findings

• Once the poison sample is retrieved, the LVLM is highly likely to output the target response, so retrieval is the primary bottleneck.
• Random crop of the poison image still leaves 45.20% PSR on Places-365.

Limitations

• Assumes the attacker can inject a poison sample into the knowledge database and, for optimization, access to the retriever (white-box).
• Attack success drops with larger retrieval number K, and with K=8 a single poison sample yields only 51.78% PSR on Places-365.
• Evaluated only on two retrievers, two 7B LVLMs and three image classification-derived datasets.

Experimental setup

Datasets: OVEN-Wiki WebQA ImageNet-1k Places-365 Country-211

Benchmarks: ImageNet-1k Places-365 Country-211

Metrics: RSR-1 RSR-K ARD PSR

Baselines: PoisonedEye-B PoisonedEye-S RoCLIP

Builds on: PoisonedRAG signed gradient descent (FGSM-style) RoCLIP

Closest papers by meaning

• Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment (CVPR 2024 · similarity 0.87)
• On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains (ICML 2025 · similarity 0.86)
• BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.85)
• Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models (NeurIPS 2024 · similarity 0.85)
• Backdooring Vision-Language Models with Out-Of-Distribution Data (ICLR 2025 · similarity 0.85)
• MP-Nav: Enhancing Data Poisoning Attacks against Multimodal Learning (ICML 2025 · similarity 0.85)