Research map / Probabilistic and scientific modeling / Treatment effects

Causal Evaluation of Membership Inference Attacks

NeurIPS 2026 · Probabilistic and scientific modeling › Treatment effects · approach: Proximal and deep causal effect estimation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Frames membership inference evaluation as causal inference, deriving causal analogues of MIA metrics and consistent estimators for multi-run, one-run (interference, stability-based) and zero-run (distribution shift, propensity-score IPW/AIPW) regimes.

Problem

Standard MIA evaluation requires costly retraining; one-run and zero-run alternatives are used but their statistical validity is unclear, suffering from interference between jointly included points and confounding from distribution shift between member and non-member data.

Approach

Defines memorization as the causal effect of including a data point in training, identifies interference (one-run) and confounding (zero-run) as bias sources, and proposes estimators: difference-in-means for one-run under algorithmic stability assumptions, and propensity-score IPW/AIPW corrections for zero-run.

Key results

• On CIFAR-10 ResNet-16 zero-run, naive AUC ≈0.72 drops to ≈0.63 with propensity correction, close to IID baseline ≈0.65.
• On fine-tuned Pythia-70m, naive MIA AUC > 0.95 under distribution shift, while propensity-corrected AUC ≈0.6 matches the IID reference.
• On Pythia-12b, distribution shift inflates ROC for arxiv, github, pubmed_central; causal correction largely removes the signal in those corpora.

Surprising findings

• Naive zero-run evaluation on DP-SGD synthetic experiments violates the ROC upper bound implied by (epsilon, delta)-DP, while corrected versions respect it.
• Causal correction largely removes the MIA signal on pretrained Pythia-12b for corpora with large distribution shift, suggesting little true memorization there.

Limitations

• One-run and zero-run consistency rely on algorithmic stability assumptions (error and uniform training stability) that hold cleanly only for convex/regularized or interpolating settings.
• Zero-run correction requires overlap and a well-estimated propensity score; IPW becomes unstable when propensity scores approach 1.
• Experiments are limited in scale (small models like Pythia-70m for the controlled setting) and rely on proxy member/non-member splits.

Experimental setup

Datasets: CIFAR-10 White House press statement titles Pile-derived text corpora (arxiv, github, pubmed_central, hackernews, wikipedia, pile_cc)

Benchmarks: CIFAR-10 Pythia-12b membership benchmark corpora (six text datasets) White House press statements (fine-tuned Pythia-70m)

Metrics: AUC ROC curves membership advantage (ATE) TPR at fixed FPR

Baselines: naive zero-run MIA evaluation multi-run MIA evaluation IID zero-run baseline

Builds on: potential outcomes framework algorithmic stability inverse probability weighting AIPW doubly robust estimation

Closest papers by meaning

• Assessing Per-Sample Membership Inference Vulnerability without Retraining (NeurIPS 2026 · similarity 0.90)
• Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration (NeurIPS 2024 · similarity 0.89)
• Exploring the limits of strong membership inference attacks on large language models (NeurIPS 2025 · similarity 0.88)
• Low-Cost High-Power Membership Inference Attacks (ICML 2024 · similarity 0.88)
• Impact of Dataset Properties on Membership Inference Vulnerability of Deep Transfer Learning (NeurIPS 2025 · similarity 0.88)
• Estimating Model-Level Membership Inference Vulnerability Without Reference Models (NeurIPS 2026 · similarity 0.87)