Research map / Vision-language and multimodal / Visual question answering
Extracting Training Data From Document-Based VQA Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows document VQA models (Donut, Pix2Struct, PaLI-3) can regurgitate training answers, including singly-occurring PII, when the answer is removed from the image, and proposes Extraction Blocking as a defense.
Problem
Document-based VQA models may memorize training question-answer pairs, including PII repeated only once, and reveal them when the answer region is removed from the document image, posing a privacy risk.
Approach
Defines extractability of an answer from a partial context (document image with the answer whited out plus the training question), separates memorization from generalization using a baseline model trained without the canaries, and proposes Extraction Blocking: adding training samples with the answer removed and the target 'ANSWER NOT PRESENT'.
Key results
- Donut at highest training resolution has |M|=63 memorized extractable answers, rising to 756 at the lowest resolution.
- Extraction Blocking reduces extractable memorized samples |M| to 0 for Pix2Struct-B, Pix2Struct-L and PaLI-3, and to 2 for Donut, while ANLS increases (+1.2 to +3.4).
- PaLI-3 at high resolution shows most extractable answers arise from generalization rather than memorization, with ANLS 87.6 on test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add answer-removed image/question training examples labeled 'ANSWER NOT PRESENT' | Donut · DocVQA canaries | memorized extractable samples |M|: 38 -> 2 | Helped |
| add answer-removed image/question training examples labeled 'ANSWER NOT PRESENT' | Pix2Struct-Base · DocVQA canaries | memorized extractable samples |M|: 40 -> 0 | Helped |
| add answer-removed image/question training examples labeled 'ANSWER NOT PRESENT' | Pix2Struct-Large · DocVQA canaries | memorized extractable samples |M|: 37 -> 0 | Helped |
| add answer-removed image/question training examples labeled 'ANSWER NOT PRESENT' | PaLI-3 · DocVQA canaries | memorized extractable samples |M|: 4 -> 0 | Helped |
| add answer-removed image/question training examples labeled 'ANSWER NOT PRESENT' | DocVQA models · DocVQA held-out validation | ANLS: +1.2; +3.4; +2.1; +1.5 | Helped |
| paraphrase the question at inference time | Donut · DocVQA canaries | memorized extractable samples |M|: 38 -> 26 | Helped |
| paraphrase the question at inference time | Donut · DocVQA held-out validation | ANLS: -12.5 | Hurt |
| decrease training image resolution (ablation) | Donut · DocVQA canaries | memorized extractable samples |M|: 63 -> 109 -> 168 -> 756 | Hurt |
| remove all text from the input image (ablation) | Donut · DocVQA canaries | memorized extractable samples |M|: 26 | Helped |
| remove all text from the input image (ablation) | Pix2Struct · DocVQA canaries | memorized extractable samples |M|: about 94 -> 27 | Helped |
| use a paraphrased training question (ablation) | Donut and Pix2Struct · DocVQA canaries | extractable answers | Helped |
| perturb image brightness (ablation) | DocVQA models · DocVQA canaries | extractable answers | Helped |
Surprising findings
- Extraction Blocking, a training-set augmentation, improves ANLS while reducing extractable memorization, whereas PR/AR and ITP lower ANLS.
- Extraction can succeed from the question alone with an unrelated image in a few cases, while image-only extraction yields no answers.
Limitations
- Extraction Blocking is a heuristic and may not stop more sophisticated attacks.
- EB may create a membership inference side-channel via the 'ANSWER NOT PRESENT' response.
- Extractability counts depend on OCR-based answer removal and manual curation of canaries.
Experimental setup
Datasets: DocVQA C4
Benchmarks: DocVQA
Metrics: extractable sample count |E| memorized count |M| ANLS memorization score simplicity score PII count
Baselines: Inference Time Paraphrasing Prepending/Appending Random String Donut Pix2Struct-Base Pix2Struct-Large PaLI-3
Builds on: Counterfactual memorization Memorization and simplicity scores Pix2Struct Donut PaLI-3
Closest papers by meaning
- Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models (ICML 2025 · similarity 0.88)
- DocMIA: Document-Level Membership Inference Attacks against DocVQA Models (ICLR 2025 · similarity 0.87)
- Private Attribute Inference from Images with Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- Quantifying and Analyzing Entity-Level Memorization in Large Language Models (AAAI 2024 · similarity 0.85)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.85)
- Déjà Vu Memorization in Vision–Language Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.