Research map / Adversarial robustness and security / Attacks on text and graphs

MP-Nav: Enhancing Data Poisoning Attacks against Multimodal Learning

ICML 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Clean-label data poisoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MP-Nav, a plug-and-play concept- and instance-level selection module that improves data poisoning attacks (AtoB for text-image retrieval, ShadowCast for VQA) by choosing semantically similar concept pairs and robust instances.

Problem

Existing data poisoning attacks on multimodal models pick concepts and poisoned instances randomly, which yields suboptimal misassociation and can fail when benign instances dilute the poison.

Approach

Use embedding-space similarity between concept means to choose source/target concept pairs, and rank instances by cosine proximity to concept centers to pick robust instances to poison, as a plug-and-play layer over existing attacks.

Key results

• On COCO, MP-Nav (Con.+Ins.) raises AtoB Hit@10 from 0.505 to 0.677 and reduces Min Rank from 15.705 to 14.602.
• On Flickr-PASCAL, MP-Nav consistently outperforms the random baseline across Hit@K metrics for sheep-to-cow vs sheep-to-aeroplane misassociation, though the paper notes gains are not reliably definitive.
• On ShadowCast Biden-Trump with a limited attacker budget (0-20 poisoned instances), MP-Nav instance selection significantly raises attack success rate; poisoned models keep GQA utility near clean (about 59.5).

Measured effects reported in the paper

Surprising findings

• The authors state MP-Nav's concept and instance selections cannot definitively improve poisoning attacks, with large uncertainty from training factors.
• Benign target-concept instances in training substantially dilute poisoned instances, so the ShadowCast attack needs instance-level selection to remain effective.

Limitations

• Gains from concept and instance selection are inconsistent and within large variance on several settings.
• Evaluation covers only two attacks (AtoB, ShadowCast) and CLIP ViT-B/32 plus LLaVA-1.5 backbones.

Experimental setup

Datasets: COCO Flickr30k PASCAL sentences MiniGPT4 dataset Food101 GQA

Benchmarks: COCO Flickr-PASCAL GQA

Metrics: Hit@1 Hit@5 Hit@10 Min Rank R@1/R@2/R@10 text and image retrieval attack success rate

Baselines: AtoB (Yang et al., 2023) random selection ShadowCast (Xu et al., 2024) random selection

Builds on: AtoB ShadowCast PGD adversarial perturbation CLIP fine-tuning

Closest papers by meaning

• Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (NeurIPS 2026 · similarity 0.86)
• Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models (NeurIPS 2024 · similarity 0.85)
• On the Robustness of Large Multimodal Models Against Image Adversarial Attacks (CVPR 2024 · similarity 0.85)
• PoisonedEye: Knowledge Poisoning Attack on Retrieval-Augmented Generation based Large Vision-Language Models (ICML 2025 · similarity 0.85)
• ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training (NeurIPS 2025 · similarity 0.85)
• From Trojan Horses to Castle Walls: Unveiling Bilateral Data Poisoning Effects in Diffusion Models (NeurIPS 2024 · similarity 0.84)