Research map / Large language models / Language, safety and interpretability
Modular Pretraining Enables Access Control
Ethan Roland, Murat Cubuktepe, Erick Martinez, Stijn Servaes, Keenan Pepper, Michael Vaiana, Diogo S de Lucena, Judd Rosenblatt, Addie Foote, Cem Anil, Alex Cloud
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GRAM instead of data filtering | GRAM · Simple Stories | retain compute ratio: 0.96 -> 0.95 | No clear effect |
| add a shared always-active core MLP (GRAM vs DEMix) (ablation) | GRAM · Simple Stories | retain compute ratio: 0.183 -> 0.952 | Improved |
| use GRAM instead of FT-LoRA | GRAM · Simple Stories | core compute ratio: 0.841 -> 0.938 | Improved |
| use GRAM instead of MaxEnt unlearning | GRAM · Simple Stories | elicited forget compute ratio: 0.983 -> 0.855 | Improved |
| use GRAM instead of data filtering with 50% labeled data | GRAM · Dual-use 800M, 50% labeled | forget compute ratio: 0.85 ± 0.01 -> 0.60 ± 0.01 | Improved |
| use GRAM instead of FT-LoRA with multiple modules active | GRAM · Dual-use 800M, all auxiliary active | retain compute ratio | Improved |
| increase number of auxiliary categories from 4 to 20 (ablation) | GRAM · Simple Stories | retain compute ratio: 4 -> 20 categories | No clear effect |
| increase number of auxiliary categories from 4 to 20 (ablation) | GRAM · Simple Stories | forget compute ratio | Hurt |
| scale model from 50M to 5B parameters | GRAM and data filtering · Dual-use scaling (Virology retained) | forget compute ratio | Improved |
| use GRAM instead of data filtering at 5B | GRAM · Dual-use scaling 5B | retain (virology) compute ratio | Mixed |
| use MaxEnt post-hoc unlearning instead of GRAM | MaxEnt · Simple Stories | forget compute ratio: 0.766 -> 0.840 | Hurt |
Closest papers by meaning
- On the creation of narrow AI: hierarchy and nonlocality of neural network skills (NeurIPS 2025 · similarity 0.83)
- Stress-Testing Capability Elicitation With Password-Locked Models (NeurIPS 2024 · similarity 0.83)
- Activation Space Interventions Can Be Transferred Between Large Language Models (ICML 2025 · similarity 0.83)
- Preparing Lessons for Progressive Training on Language Models (AAAI 2024 · similarity 0.82)
- The Best Instruction-Tuning Data are Those That Fit (NeurIPS 2025 · similarity 0.82)
- The Elicitation Game: Evaluating Capability Elicitation Techniques (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
