Research map / Federated learning and privacy / Privacy attacks and secure learning
Gradient Inversion of Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GI-DQA, a gradient inversion attack that reconstructs private document text in federated learning of vision-language document QA models (Donut, LayoutLMv3), using template priors and a defense called Safe Template.
Problem
Gradient inversion attacks on multimodal vision-language models, specifically document visual question answering (DQA) under federated learning, are largely unexplored; existing unimodal attacks fail to recover fine-grained document text.
Approach
Combine an analytic step that recovers question-answer tokens with an optimization-based document reconstruction that uses publicly available document templates as initialization and adds visual-text priors (Laplacian, Gaussian, channel/spatial TV) with an exponentially decaying prior scheduler, plus a combined Euclidean and cosine gradient matching loss emphasizing pre-fusion gradients.
Key results
- On Donut, GI-DQA perfectly reconstructs 70.1% of document words (binary match) vs 5.8% for APRIL and 0% for Inverting Gradients.
- On LayoutLMv3, GI-DQA achieves 82.1% binary match and PSNR 22.713, while DLG and iDLG perform at random-baseline level.
- Ablation shows reconstruction works equally well with random QA tokens (71.9% binary), indicating reliance on exact token identities and positions rather than semantics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace prior inversion attacks with GI-DQA | DQA gradient inversion on Donut · PFL-DocVQA | Binary: 5.8% -> 70.1% | Helped |
| replace prior inversion attacks with GI-DQA | DQA gradient inversion on LayoutLMv3 · PFL-DocVQA | Binary: 0.6% -> 82.1% | Helped |
| add channel-wise TV prior (ablation) | GI-DQA document reconstruction · PFL-DocVQA / Donut | Binary: 42.6% -> 61.0% | Helped |
| add spatial TV prior (ablation) | GI-DQA document reconstruction with channel-wise TV · PFL-DocVQA / Donut | PSNR: 22.555 -> 24.540 | Helped |
| add visual text prior (ablation) | GI-DQA document reconstruction with TV priors · PFL-DocVQA / Donut | Binary: 70.2% -> 70.6% | Helped |
| add Gaussian prior (ablation) | GI-DQA document reconstruction with TV and visual text priors · PFL-DocVQA / Donut | PSNR: 23.855 -> 24.194 | Helped |
| use only gradient matching loss, without auxiliary priors (ablation) | GI-DQA document reconstruction · PFL-DocVQA / Donut | Binary: 61.0% -> 42.6% | Hurt |
| provide both the question and answer rather than withholding either (ablation) | GI-DQA document reconstruction · PFL-DocVQA / Donut | Binary: 1.0% -> 70.1% | Helped |
| replace the original QA text with random tokens of the same length and structure (ablation) | GI-DQA document reconstruction · PFL-DocVQA / Donut | Binary: 70.1% -> 71.9% | No clear effect |
| attack at later rather than earlier training stages | GI-DQA · PFL-DocVQA | reconstruction quality | Hurt |
| use post-fusion gradients instead of pre-fusion gradients | GI-DQA on Donut · PFL-DocVQA / Donut | reconstruction quality | Hurt |
| use post-fusion gradients instead of pre-fusion gradients | GI-DQA on LayoutLMv3 · PFL-DocVQA / LayoutLMv3 | reconstruction success | Hurt |
Surprising findings
- Replacing the question-answer pair with random tokens of the same length yields nearly identical reconstruction quality.
- Post-fusion gradients are much less revealing; on LayoutLMv3 reconstruction fails entirely when only post-fusion gradients are used, suggesting multimodal fusion acts as an implicit safeguard.
Limitations
- Assumes the adversary has access to public document templates and the exact token identities and positions used by the client.
- Requires first recovering question-answer pairs via analytic methods; document reconstruction is infeasible without them.
- Evaluated only on invoice DQA data (PFL-DocVQA subset of 395 documents) and two base-size models.
Experimental setup
Datasets: PFL-DocVQA
Benchmarks: PFL-DocVQA
Metrics: PSNR FFT2D cosine similarity MSE exact match (binary) fuzz ratio
Baselines: DLG iDLG Inverting Gradients APRIL
Builds on: DLG iDLG Inverting Gradients APRIL GradViT
Closest papers by meaning
- Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models (ICLR 2025 · similarity 0.87)
- A New Federated Learning Framework Against Gradient Inversion Attacks (AAAI 2025 · similarity 0.86)
- Enhancing Storage and Computational Efficiency in Federated Multimodal Learning for Large-Scale Models (ICML 2024 · similarity 0.86)
- Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning (CVPR 2025 · similarity 0.85)
- Concealing Sensitive Samples against Gradient Leakage in Federated Learning (AAAI 2024 · similarity 0.85)
- Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers, and Gradient Clipping (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.