Research map / Adversarial robustness and security / Attacks on text and graphs
Variance-Based Membership Inference Attacks Against Large-Scale Image Captioning Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes two shadow-free, image-only membership inference attacks (MVTA and C-WSA) against image captioning models, using a means-of-variance metric over multiple sampled captions.
Problem
Membership inference against image captioning models is under-studied; prior attacks rely on shadow models or on ground-truth captions, which limits applicability to large-scale public models.
Approach
Memorized training images yield lower variance among embeddings of multiple top-p sampled captions; the means-of-variance (MV) metric quantifies this, enabling a zero-knowledge threshold attack (MVTA) and a confidence-thresholded weakly supervised attack (C-WSA) that use only images.
Key results
- C-WSA achieves AUC 0.91 on BLIP and 0.93 on ViT-GPT2 on MSCOCO, versus WSA's 0.84 and 0.89.
- MVTA reaches AUC 0.66 on BLIP and 0.77 on ViT-GPT2 versus CSA's 0.54 and 0.72 in the zero-knowledge setting.
- C-WSA TPR@FPR=1% is 0.38 on BLIP and 0.43 on ViT-GPT2 on MSCOCO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CSA with MVTA | Image captioning membership inference · MSCOCO (BLIP Large) | AUC: 0.5444 -> 0.6601 | Helped |
| replace CSA with MVTA | Image captioning membership inference · MSCOCO (ViT-GPT2) | AUC: 0.7207 -> 0.7727 | Helped |
| replace WSA with C-WSA | Image captioning membership inference · MSCOCO (BLIP Large) | AUC: 0.8398 -> 0.9131 | Helped |
| replace WSA with C-WSA | Image captioning membership inference · MSCOCO (ViT-GPT2) | AUC: 0.8931 -> 0.9335 | Helped |
| replace WSA with C-WSA | Image captioning membership inference · MSCOCO (GIT Base) | AUC: 0.8626 -> 0.9255 | Helped |
| replace WSA with C-WSA | Image captioning membership inference · TextCaps (GIT Base) | AUC: 0.8165 -> 0.8967 | Helped |
| set λ to 0.5 (ablation) | C-WSA · MSCOCO (BLIP Large) | AUC: 0.9041 -> 0.9131 | Helped |
| set λ to 0.5 (ablation) | C-WSA · MSCOCO (ViT-GPT2) | AUC: 0.9235 -> 0.9304 | Helped |
| add a confidence threshold during attack-classifier training (ablation) | C-WSA · pseudo-member training samples | average probability assigned to mislabeled non-members | Helped |
| use top-p sampling instead of beam search (ablation) | MVTA and C-WSA · MSCOCO | AUC | Helped |
| increase captions generated per image (ablation) | MVTA and C-WSA · MSCOCO | AUC | Helped |
| increase the member ratio in Dall (ablation) | C-WSA · MSCOCO | AUC | Helped |
Surprising findings
- MVTA performs near chance on TextCaps with GIT (AUC 0.50), indicating the zero-knowledge attack fails on that dataset.
Limitations
- Evaluated only on three public captioning models and two datasets; diffusion and other generative models not covered.
- MVTA performance is weak on some settings (e.g., TextCaps with GIT).
- Attack depends on a non-member dataset from the same distribution and on a relatively large number of sampled captions per image.
Experimental setup
Datasets: MSCOCO FLICKR30k TextCaps IAPR TC-12 NoCaps
Benchmarks: MSCOCO TextCaps
Metrics: AUC ACC TPR@FPR=1%
Baselines: CSA WSA MVTA
Builds on: WSA (weakly supervised attack) cosine similarity attack (CSA) CLIP
Closest papers by meaning
- Membership Inference Attacks against Large Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Vid-SME: Membership Inference Attacks against Large Video Understanding Models (NeurIPS 2025 · similarity 0.87)
- Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration (NeurIPS 2024 · similarity 0.87)
- Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probing (NeurIPS 2025 · similarity 0.86)
- Backdooring Vision-Language Models with Out-Of-Distribution Data (ICLR 2025 · similarity 0.86)
- Private Attribute Inference from Images with Vision-Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.