Research map / Adversarial robustness and security / Attacks on text and graphs

Estimating Model-Level Membership Inference Vulnerability Without Reference Models

NeurIPS 2026 · Adversarial robustness and security › Attacks on text and graphs · approach: Membership inference and inversion

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes estimating a model's LiRA TPR at low FPR from its train and test loss distributions alone, using LOSS TNR for heavy-tailed (image) models and LOSS AUC for symmetric (GPT-2) models, with no reference models.

Problem

Measuring model-level vulnerability to state-of-the-art reference-model membership inference (LiRA) requires training many costly reference models, which limits practical privacy auditing.

Approach

Decomposes LiRA's per-sample score into a variance-ratio term and a residual mean-shift term; the regime (heavy-tailed vs symmetric loss distributions) determines whether a tail statistic (LOSS TNR at a matched FNR) or a mean-separation statistic (LOSS AUC) predicts LiRA TPR, with a KS-based diagnostic choosing between them.

Key results

• LOSS TNR predicts LiRA TPR@FPR=1e-3 with RMSE 0.036 (R^2 0.944) across 10 image architectures and 4 datasets, beating RMIA with 2 reference models (RMSE 0.046).
• LOSS AUC predicts LiRA TPR with RMSE 0.018 across five GPT-2 sizes from 10M to 1B parameters.
• Under leave-one-architecture-out, RMSE is 0.038 vs 0.036 in-sample; leave-one-dataset-out RMSE is 0.064.

Surprising findings

• Train-test accuracy gap is a weak predictor compared with LOSS TNR, because it captures only mean shifts and misses the asymmetric non-member tail.

Limitations

• Evaluated only against reference-model attacks (LiRA, RMIA, Attack R); transferability to other attack families and reconstruction/inversion threats is untested.
• LLM evaluation limited to five GPT-2 models with LiRA and eight Pythia models using RMIA as a proxy for LiRA.
• The estimator is empirically calibrated with a linear fit, and the regime-selection threshold tau≈0.5 is heuristic.

Experimental setup

Datasets: MNIST CIFAR-10 CIFAR-100 CINIC-10 C4 The Pile

Benchmarks: LiRA TPR@FPR=1e-3 across 10 image architectures and 4 datasets LiRA TPR across five GPT-2 sizes Pythia RMIA TPR

Metrics: RMSE R^2 MAE Pearson correlation TPR@FPR

Baselines: RMIA LT-IQR LOSS AUC train-test accuracy gap KS distance Attack R

Builds on: LiRA RMIA LOSS attack

Closest papers by meaning

• Low-Cost High-Power Membership Inference Attacks (ICML 2024 · similarity 0.89)
• Assessing Per-Sample Membership Inference Vulnerability without Retraining (NeurIPS 2026 · similarity 0.88)
• Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration (NeurIPS 2024 · similarity 0.88)
• Exploring the limits of strong membership inference attacks on large language models (NeurIPS 2025 · similarity 0.88)
• Impact of Dataset Properties on Membership Inference Vulnerability of Deep Transfer Learning (NeurIPS 2025 · similarity 0.88)
• Causal Evaluation of Membership Inference Attacks (NeurIPS 2026 · similarity 0.87)