Research map / Federated learning and privacy / Privacy attacks and secure learning

Privacy Backdoors: Stealing Data with Corrupted Pretrained Models

ICML 2024 · Federated learning and privacy › Privacy attacks and secure learning · approach: Membership inference and auditing · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that a malicious provider can tamper with a pretrained model's weights so that finetuning writes exact training inputs into the weights ("data traps"), enabling near-perfect data reconstruction, membership inference, and tight attacks on DP-SGD.

Problem

Finetuning downloaded pretrained models creates a supply-chain risk: an attacker who tampers with the pretrained weights may steal the private finetuning data, including data protected by differential privacy.

Approach

Single-use "data traps" in linear units that capture one training input and then shut off via the gradient they themselves produce, so captured inputs survive a full finetuning run; extended to transformers via keyed backdoors (position and sequence identifiers) and stabilized layer norms and GELU handling.

Key results

• Backdoored ViT-B/32 finetuned on Caltech 101 reaches 83% test accuracy and recovers recognizable full images; backdoored BERT on TREC-50 reaches 78% and reconstructs sentences near-perfectly.
• Black-box membership inference via backdoor units gives 100% accuracy in distinguishing training members.
• End-to-end empirical privacy leakage on backdoored CNNs with DP-SGD is close to the provable epsilon bound from the DP-SGD analysis.

Measured effects reported in the paper

Surprising findings

• Loose privacy budgets (e.g., epsilon above 8 or 9) considered acceptable in practice become insecure when the model provider is untrusted.

Limitations

• Only full finetuning is considered; parameter-efficient methods like LoRA are left to future work.
• Backdoors that fail to shut down can break model utility, so attacks are limited to settings with fewer backdoors than output classes; the backdoored transformers lose some test accuracy.

Experimental setup

Datasets: CIFAR-10 Oxford-IIIT Pet Caltech 101 TREC-50

Benchmarks: CIFAR-10 Oxford-IIIT Pet Caltech 101 TREC-50

Metrics: reconstruction quality test accuracy membership inference accuracy empirical privacy budget (epsilon)

Baselines: Federated Learning gradient leakage attacks DP-SGD analysis of Abadi et al.

Builds on: Federated Learning data stealing via sparse activations (Boenisch et al.) DP-SGD model stealing via critical points

Closest papers by meaning

• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models (NeurIPS 2024 · similarity 0.91)
• Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining (ICML 2024 · similarity 0.88)
• Attack-Aware Noise Calibration for Differential Privacy (NeurIPS 2024 · similarity 0.87)
• Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakage (AAAI 2025 · similarity 0.87)
• Leak and Learn: An Attacker's Cookbook to Train Using Leaked Data from Federated Learning (CVPR 2024 · similarity 0.87)
• Provable Robustness against Backdoor Attacks via the Primal-Dual Perspective on Differential Privacy (NeurIPS 2026 · similarity 0.86)