Research map / Large language models / Language, safety and interpretability
(Token-Level) \textbf{InfoRMIA}: Stronger Membership Inference and Privacy Assessment for LLMs
Jiashu Tao, Reza Shokri
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace RMIA's thresholded population-count score with continuous InfoRMIA information-theoretic score | RMIA · AG News, |Z|=100 | TPR@0.1%FPR: 0.00% -> 12.0% | Improved |
| replace RMIA with InfoRMIA | RMIA · CIFAR-10, |Z|=1000 | AUC: 0.8229 -> 0.8330 | Improved |
| replace RMIA with InfoRMIA | RMIA · Purchase100, |Z|=1000 | AUC: 0.5311 -> 0.5754 | Improved |
| increase population size |Z| for RMIA from 100 to 1000 (ablation) | RMIA · AG News | AUC: 0.8574 -> 0.8766 | Improved |
| increase population size |Z| for InfoRMIA from 100 to 1000 (ablation) | InfoRMIA · AG News | AUC: 0.8784 -> 0.8784 | No clear effect |
| use token-level InfoRMIA averaged over tokens instead of sequence-level InfoRMIA (ablation) | InfoRMIA · finetuned GPT-2 sequence-level MIA | AUC: AG News 4 ep 0.945 -> 0.942; ai4privacy 4 ep 0.822 -> 0.804; TPR AG News 4 ep 16.2% -> 20.6% | Mixed |
| aggregate token scores by min-k instead of averaging (ablation) | token-level InfoRMIA · MIMIR pretrained Pythia | TPR at low FPR | Hurt |
| use min-k instead of averaging for aggregation (ablation) | token-level InfoRMIA · MIMIR pretrained Pythia | AUC | Improved |
| use step-1 Pythia-160M checkpoint as reference model rather than later checkpoints (ablation) | token-level InfoRMIA · MIMIR pretrained Pythia | membership inference performance | Improved |
| evaluate private tokens only vs all tokens at sequence level | InfoRMIA on ai4privacy · ai4privacy fine-tuned GPT-2 | average membership score | Hurt |
Closest papers by meaning
- Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.89)
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.89)
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.89)
- Low-Cost High-Power Membership Inference Attacks (ICML 2024 · similarity 0.89)
- The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text (ICML 2025 · similarity 0.89)
- Quantifying and Analyzing Entity-Level Memorization in Large Language Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
