Research map / Adversarial robustness and security / Attacks on text and graphs

Medical Multimodal Model Stealing Attacks via Adversarial Domain Alignment

AAAI 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Membership inference and inversion · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ADA-Steal, a data-free model stealing attack that replicates a black-box medical MLLM (CheXagent) for radiology report generation by enriching reports with an open LLM oracle and aligning non-medical query images via adversarial perturbations.

Problem

Stealing the functionality of black-box medical multimodal LLMs for radiology report generation, where the attacker has no medical data and the victim's output space (text vocabulary) is large, making classification-era stealing methods ineffective.

Approach

Use an open LLM (Zephyr-7B) without image input to generate diversified, medically relevant target reports, then craft FGSM adversarial perturbations on natural query images using the attacker model so that the victim's responses to perturbed images become more diverse and clinically relevant, iteratively aligning the attacker query distribution with the medical image domain.

Key results

• On MIMIC-CXR, IDEFICS+ADA-Steal reaches RG-L 23.2 vs. victim CheXagent 26.5 and IDEFICS+Knockoff 20.2, with RadGraph 15.6 vs. 20.7 victim.
• On IU X-Ray, IDEFICS+ADA-Steal reaches RG-L 29.8 and Rad-S 19.4 versus victim 32.2 and 26.1, outperforming Knockoff (22.9 and 14.0).
• Using CIFAR-100 query images, attacks produce near-victim clinical efficacy; using random Gaussian noise as query images fails to steal (RG-L 7.0 on MIMIC-CXR, below the original IDEFICS 14.6).

Measured effects reported in the paper

Surprising findings

• Using MIMIC-CXR training images as the attacker query set performs worse than CIFAR-100 queries, attributed to adversarial perturbations distorting real CXR distributions.
• CheXbert-Score is unreliable for out-of-distribution text: a corpus of repeated 'The' or empty strings scores comparably or higher on accuracy and F1 than the attack reports, so the metric was dropped.

Limitations

• Evaluated only on deterministic victims using beam search decoding.
• Does not evaluate against model stealing defenses.
• Reports from the oracle LLM are not grounded in images and may be inaccurate.

Experimental setup

Datasets: CIFAR-100 MIMIC-CXR IU X-Ray

Benchmarks: MIMIC-CXR IU X-Ray

Metrics: ROUGE-L BERTScore RadGraph-Score GPT-4 pairwise evaluation CheXbert F1

Baselines: IDEFICS Knockoff Nets CheXagent*

Builds on: Knockoff Nets FGSM Bert-Thieves data-free knowledge distillation

Closest papers by meaning

• Medical MLLM Is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Models (AAAI 2025 · similarity 0.87)
• Prompt2Perturb (P2P): Text-Guided Diffusion-Based Adversarial Attack on Breast Ultrasound Images (CVPR 2025 · similarity 0.84)
• Fragments to Facts: Partial-Information Fragment Inference from LLMs (ICML 2025 · similarity 0.83)
• CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs (NeurIPS 2025 · similarity 0.83)
• Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.83)
• Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images (ICLR 2025 · similarity 0.82)