Research map / Vision-language and multimodal / Multimodal LLMs
Membership Inference Attacks against Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VL-MIA, the first membership inference benchmark for large vision-language models, plus a cross-modal token-level image MIA pipeline and the MaxRényi-K% entropy-based metric.
Problem
Detecting whether individual images or texts were used to train large vision-language models is unresolved due to lack of standardized MIA datasets and suitable methods, since image inputs have no ground-truth tokens for target-based attacks.
Approach
Generate image descriptions with the VLLM, then re-feed image, instruction and description to extract logit slices per modality; propose MaxRényi-K%, a target-free score averaging the top-K% Rényi entropies of next-token distributions, and a target-based ModRényi variant.
Key results
- On GPT-4 image MIA, MaxRényi-K% (alpha=0.5, K=100%) reaches AUC 0.815 on VL-MIA/DALL-E.
- On LLaVA image MIA, Rényi (alpha=0.5, K=100%) achieves AUC 0.702 on VL-MIA/Flickr image slice, outperforming target-based baselines such as Min-K% Prob.
- On LLM pre-training text (WikiMIA, LLaVA), target-free MaxRényi at large alpha consistently outperforms target-based methods, supporting the hypothesis that model change favors target-free metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use target-free MaxRényi-K% rather than target-based metrics for image MIA | VLLM image MIA pipeline · VL-MIA image MIA | AUC | Helped |
| Set Rényi order α=0.5 rather than larger α values | MaxRényi-K% image MIA · VL-MIA/Flickr and VL-MIA/DALL-E | AUC | Helped |
| Use target-based MIA rather than target-free MIA | VLLM instruction-tuning text detection · VL-MIA/Text | AUC | Helped |
| Use large-α target-free MIA rather than target-based MIA | LLaVA pre-training text detection · WikiMIA | AUC | Helped |
| Increase generated description length (ablation) | LLaVA 1.5 image MIA with MaxRényi-10% · VL-MIA/Flickr | AUC | Helped |
| Use max_new_tokens beyond 128 rather than stop at 128 (ablation) | LLaVA 1.5 image MIA with MaxRényi-10% · VL-MIA/Flickr | AUC | No clear effect |
| Corrupt member images (ablation) | LLaVA image MIA with MaxRényi-K% (α=0.5) · VL-MIA/Flickr corrupted images | AUC | Hurt |
| Apply JPEG or Motion_Blur corruption rather than Brightness or Snow (ablation) | LLaVA image MIA · VL-MIA/Flickr corrupted images | AUC | Hurt |
| Change the instruction text (ablation) | LLaVA image MIA pipeline · VL-MIA/Flickr | AUC | No clear effect |
| Use MinRényi-K% rather than MaxRényi-K% (ablation) | LLaVA pre-training text MIA · WikiMIA | AUC | Hurt |
| Expand the VL-MIA/Flickr and VL-MIA/Text datasets to 2000 samples (ablation) | VL-MIA experiments · VL-MIA/Flickr and VL-MIA/Text | AUC | No clear effect |
Surprising findings
- Target-free metrics outperform target-based metrics for image MIAs, while target-based metrics are better for instruction-tuning text.
- Instruction slices yield unstable AUCs, sometimes below 0.5 for target-based metrics.
Limitations
- GPT-4 experiments require approximating the full distribution by assuming a 32000-token vocabulary with uniform remaining probability.
- Main experiments use small datasets of about 600 samples per set due to compute constraints.
- Grey-box assumption requires access to output logits of the target model.
Experimental setup
Datasets: MS COCO LAION-CCS LLaVA-Instruct-150K WikiMIA MNIST EMNIST Flickr
Benchmarks: VL-MIA/Flickr VL-MIA/DALL-E VL-MIA/Text WikiMIA
Metrics: AUC TPR@5%FPR
Baselines: Perplexity Min-K% Prob Aug-KL Loss attack ppl/zlib ppl/lowercase Max_Prob_Gap Entropy
Builds on: Min-K% Perplexity Aug-KL Modified Entropy WikiMIA
Closest papers by meaning
- Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probing (NeurIPS 2025 · similarity 0.92)
- Variance-Based Membership Inference Attacks Against Large-Scale Image Captioning Models (CVPR 2025 · similarity 0.91)
- MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector (AAAI 2025 · similarity 0.91)
- Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration (NeurIPS 2024 · similarity 0.90)
- Vid-SME: Membership Inference Attacks against Large Video Understanding Models (NeurIPS 2025 · similarity 0.90)
- DF-MIA: A Distribution-Free Membership Inference Attack on Fine-Tuned Large Language Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.