Research map / Architectures and efficiency / Pruning and efficient tuning
Provable unlearning in topic modeling and downstream tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Provides the first provable (epsilon,delta) unlearning guarantees for topic models in the pre-training and fine-tuning pipeline, with deletion capacity bounds that improve when the model is adapted to a sparse downstream task.
Problem
Verifiable, provably correct machine unlearning for the pre-training and fine-tuning paradigm, where existing provable guarantees are mostly limited to supervised convex settings.
Approach
Adapts (epsilon,delta)-indistinguishability to topic models: a Newton-step update on per-word topic coefficients after removing forget documents, followed by Gaussian-mechanism noise, with deletion capacity analysed for both the base model and a head-tuned downstream model that releases only the fused matrix B = A w, so unlearning from the fine-tuned model needs no change to the base model.
Key results
- Unlearning from a pre-trained topic model is utility-preserving for deletion capacity m_U = Omega~(m / (r^2 sqrt(nr))) documents, with computation independent of dataset size.
- After head tuning on a task depending on a subset of topics with mass q, deletion capacity grows to c'·mq/(r·sqrt(nr)), i.e. easier to unlearn pre-training data from the fine-tuned model without modifying the base model.
- Unlearning updates run in O(nr^2) time versus retraining costs that grow with m and n^2 r, per the table in the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Unlearn pre-training data from the fine-tuned task model rather than requiring unlearning from the base model | Topic models adapted to a downstream classification task · Downstream topic classification | deletion capacity: T_{\epsilon,\delta}^{\learnhead,\unlearnhead}(m) \geq c'\cdot \frac{mq}{r \sqrt{nr}}; T_{\epsilon,\delta}^{\mathcal{A},\ | Helped |
| Reuse learned anchor words instead of identifying them again | Base-model unlearning · Anchor word identification | runtime: \gO(n^2 + nr/{\epsilon}_0^2) -> \gO(1) | Helped |
| Update co-occurrence frequencies for the forget set rather than recomputing them from the full dataset | Base-model unlearning · Co-occurrence matrix computation | runtime: \gO(m) -> \gO(m_U) | Helped |
| Use a projected Newton step to update topic coefficients instead of recovering topics from scratch | Base-model unlearning · Topic recovery from anchors | runtime: \gO(n^2r + nr^2/{\epsilon}_0^2) -> \gO(nr^2) | Helped |
Surprising findings
- Unlearning pre-training data is easier (larger deletion capacity) when the model is adapted to a downstream task that depends on only a subset of topics, than for the base model.
Limitations
- Analysis is restricted to topic models, which cannot capture autoregressive or other features of modern language models.
- Guarantees rely on strong structural assumptions such as p-separability, anchor words, and strong convexity of the head loss.
- Naive setting that releases A and w separately requires retraining the head and erodes utility through repeated noising.
Experimental setup
Metrics: entrywise error from ground-truth topic matrix (epsilon,delta)-indistinguishability deletion capacity
Baselines: retraining from scratch
Builds on: anchor-word topic model learning (Arora et al.) (epsilon,delta)-certified unlearning Gaussian mechanism
Closest papers by meaning
- A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.91)
- MUSE: Machine Unlearning Six-Way Evaluation for Language Models (ICLR 2025 · similarity 0.90)
- In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.90)
- Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting (NeurIPS 2025 · similarity 0.89)
- RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models (NeurIPS 2024 · similarity 0.89)
- Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.