Research map / Large language models / Language, safety and interpretability
Hubble: a Model Suite to Advance the Study of LLM Memorization
Johnny Wei, Ameya Godbole, Mohammad Aflah Khan, Ryan Yixiang Wang, Xiaoyuan Zhu, James Flemings, Nitya Kashyap, Krishna P. Gummadi, Willie Neiswanger, Robin Jia
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on larger corpus (500B vs 100B tokens) with same inserted data (dilution) (ablation) | Hubble 8B perturbed · memorization across copyright/privacy/test-set tasks | memorization vs duplicate count | Improved |
| insert sensitive data only in first quarter of training (ablation) | Hubble 1B timing runs · memorization evals | memorization | Improved |
| insert perturbations only in last quarter of training (ablation) | Hubble 1B timing runs · memorization evals | memorization | Hurt |
| increase model size from 1B to 8B (ablation) | Hubble perturbed (500B) · memorization evals | memorization at fixed duplication | Hurt |
| mix perturbations from all three domains vs single domain (ablation) | Hubble 1B 100B perturbed · memorization evals | memorization | No clear effect |
| use popular rather than unpopular Gutenberg books (ablation) | Hubble 1B/8B · passage memorization | generative extraction | No clear effect |
| give attacker more auxiliary information (full prefix vs name only) (ablation) | PII attacks on Hubble 8B 100B · YAGO/ECtHR PII inference | attack accuracy: close to 100% at 16 duplications | Improved |
| contaminate test sets with inserted examples (ablation) | Hubble perturbed models · unseen test examples (WinoGrande, PopQA, HellaSwag) | accuracy at 0 duplicates | Mixed |
| increase duplicate count of inserted examples (ablation) | MIA methods on Hubble 8B 500B · Gutenberg Unpopular | ROC AUC: 0.539 -> 1.0 (Loss, dup 1 -> 256) | Improved |
| use MinK%++ instead of Loss at high duplication (ablation) | MIA on Gutenberg Unpopular dup=256 · Hubble 8B 500B | ROC AUC: 1.0 -> 0.949 | Hurt |
| use MinK%++ instead of Loss at low-medium duplication (ablation) | MIA on MMLU dup=16 · Hubble 8B 500B | ROC AUC: 0.857 -> 0.943 | Improved |
| apply unlearning (RMU, RR, SatImp) | Hubble 8B 500B perturbed · Gutenberg Unpopular / YAGO | unlearn vs keep set | Mixed |
Closest papers by meaning
- Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.90)
- Rethinking LLM Memorization through the Lens of Adversarial Compression (NeurIPS 2024 · similarity 0.90)
- Mitigating Memorization in Language Models (ICLR 2025 · similarity 0.90)
- Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (ICLR 2025 · similarity 0.89)
- Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects (NeurIPS 2026 · similarity 0.89)
- Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
