Research map / Adversarial robustness and security / Attacks on text and graphs
DocMIA: Document-Level Membership Inference Attacks against DocVQA Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DocMIA, auxiliary-data-free white-box and black-box membership inference attacks on document-level DocVQA models, using optimization-based distance/step/utility features clustered unsupervised, with black-box attacks via distilled proxy models.
Problem
Determining whether a single document (with multiple QA pairs) was in a DocVQA model's training set, without auxiliary data or shadow models, when outputs are autoregressive and standard logit/loss MIA features are ineffective.
Approach
Fine-tune the target model (or a LoRA/last layer, or the input image pixels) on each individual question-answer pair and use the parameter-change distance, number of optimization steps, and utility trajectory as membership features, aggregated per document and clustered with KMeans; for black-box access, distill the target into a proxy model and apply the white-box features to it.
Key results
- White-box FL attack on VT5 (PFL) reaches 66.63% accuracy and 72.40% F1, up to +11.73 F1 over ScoreLoss-UA_all.
- On DocVQA, the white-box IG attack reaches 75.17% accuracy on Donut, +4 points over the best baseline.
- Black-box proxy VT5 attack (FL) achieves TPR 23.00% at 3% FPR against Donut on DocVQA, versus 15.67% for the best Score-UA baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use optimization-based FL, FLLoRA, or IG features instead of baseline membership attacks | DocMIA white-box attacks on DocVQA models · PFL-DocVQA and DocVQA | F1: +11.73 | Helped |
| Use optimization-based DocMIA features rather than a single-step gradient attack | Donut · DocVQA | F1: up to a 10% improvement in F1 | Helped |
| Distill black-box outputs into a proxy and apply FL | Black-box DocMIA · DocVQA; Pix2Struct-B target, Donut proxy | F1: +4.88 | Helped |
| Use a proxy with the same architecture as the black-box target | Black-box DocMIA · PFL-DocVQA | membership inference performance | Helped |
| Tune the final fully connected decoder layer rather than randomly selected intermediate decoder layers (ablation) | FL and FLLoRA attacks · PFL-DocVQA and DocVQA | accuracy: 68.33 vs 61.33 | Helped |
| Use learning rate α = 10^-3 rather than other tested learning rates (ablation) | White-box DocMIA · Most model and dataset settings | attack performance | Helped |
| Combine score, optimization distance, and optimization steps instead of using score alone (ablation) | White-box DocMIA · PFL-DocVQA, VT5 | F1: 68.88 -> 71.53 | Helped |
| Use optimization distance alone instead of DocVQA score alone (ablation) | White-box DocMIA · PFL-DocVQA, Donut | F1: 67.58 -> 71.36 | Helped |
| Vary the selected layer for optimization (ablation) | FL and FLLoRA attacks · VT5 PFL, Donut DocVQA, and Pix2Struct-B DocVQA | accuracy: 67.0; 65.33; 68.33; 68.17; 61.83; 61.33; 64.0 | Mixed |
| Optimize document pixels (IG) rather than model parameters (FL) (ablation) | White-box DocMIA · PFL-DocVQA and DocVQA | F1: 79.39; 72.82 vs 72.40; 76.60 | Mixed |
Surprising findings
- Loss-threshold attack performs poorly on most DocVQA targets (under 60% accuracy) but is strongest on VT5 (DocVQA) and Donut (PFL) where overfitting is clearer.
- Attacking a proxy with a different architecture still works, indicating membership signal transfers via distillation without knowing the target architecture.
Limitations
- Assumes the adversary knows the exact training questions or can approximate them from document type.
- Performance varies widely across target models and datasets; no single baseline or variant dominates.
- Relies on public checkpoints rather than models trained by the authors for most targets.
Experimental setup
Datasets: DocVQA PFL-DocVQA
Benchmarks: DocVQA PFL-DocVQA
Metrics: accuracy F1 score TPR at 1% FPR TPR at 3% FPR
Baselines: Score-TA Score-UA Score-UA_all Loss-TA Gradient-UA ScoreLoss-UA_all Min-K% Min-K%++
Builds on: Loss-threshold membership inference Min-K% Prob Min-K%++ LoRA Score-based unsupervised MIA
Closest papers by meaning
- Extracting Training Data From Document-Based VQA Models (ICML 2024 · similarity 0.87)
- Membership Inference Attacks against Large Vision-Language Models (NeurIPS 2024 · similarity 0.85)
- Variance-Based Membership Inference Attacks Against Large-Scale Image Captioning Models (CVPR 2025 · similarity 0.85)
- Gradient Inversion of Multimodal Models (ICML 2025 · similarity 0.84)
- Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.84)
- The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.