Research map / Large language models / Language, safety and interpretability
Probing Hidden Knowledge Holes in Unlearned LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes unlearned LLMs for "knowledge holes" (benign knowledge lost during unlearning) using adjacent-keyword test generation plus RL (PPO) search with an LLM judge, showing large hidden utility losses on four unlearning methods.
Problem
Standard static benchmarks (MMLU, TruthfulQA, MT-Bench) show little utility loss after LLM unlearning, but unlearning may silently remove benign knowledge near or beyond the forget set, so existing evaluations miss these knowledge holes.
Approach
Generate test prompts in two stages: adjacent-knowledge prompts built from keywords of the forget set, and latent-knowledge prompts found by training a Llama-2-7B policy with PPO to maximize a judge-based reward for gibberish or low-quality responses from the unlearned model, followed by GPT-4o-mini validity and overlap filtering.
Key results
- On RMU unlearning (WMDP-Bio), latent-set judgescore fell from 7.747 to 1.040 with 98.7% of prompts scoring 1, while MMLU dropped only from 0.585 to 0.574.
- Adjacent-set judgescore for GAKL fell from 7.261 to 2.255 (75.2% scored 1), while MT-Bench stayed near 5.5 versus 6.7 before.
- One-shot mitigation by retraining on identified holes restored some scores but raised harmscore to 0.9 under KL and created new latent holes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply LLMU unlearning to the pretrained model | Mistral-7B-Instruct-v0.1 · Adjacent knowledge probing set (D_AP) | judgescore: 7.261 -> 3.453 | Hurt |
| Apply GAKL unlearning to the pretrained model | Mistral-7B-Instruct-v0.1 · Latent knowledge probing set (D_LP) | judgescore: 6.103 -> 1.126 | Hurt |
| Apply NPOGD unlearning to the pretrained model | Mistral-7B-Instruct-v0.1 · Adjacent knowledge probing set (D_AP) | judgescore: 7.261 -> 3.658 | Hurt |
| Apply RMU unlearning to the pretrained model | Zephyr-7B · Latent knowledge probing set (D_LP) | judgescore: 7.747 -> 1.040 | Hurt |
| Set minibatch size to 4 instead of the default (ablation) | RL probing policy · D_LP | judgescore: 1.13 -> 1.20 | No clear effect |
| Set minibatch size to 16 instead of the default (ablation) | RL probing policy · D_LP | judgescore: 1.13 -> 1.17 | No clear effect |
| Set gibberish penalty to 0.1 instead of the default (ablation) | RL probing policy · D_LP | judgescore: 1.13 -> 1.17 | No clear effect |
| Set gibberish penalty to 1.0 instead of the default (ablation) | RL probing policy · D_LP | judgescore: 1.13 -> 1.16 | No clear effect |
| Set cosine similarity reward to -0.1 instead of the default (ablation) | RL probing policy · D_LP | judgescore: 1.13 -> 1.13 | No clear effect |
| Set cosine similarity reward to -2.0 instead of the default (ablation) | RL probing policy · D_LP | judgescore: 1.13 -> 1.16 | No clear effect |
| Set BLEU reward to -0.1 instead of the default (ablation) | RL probing policy · D_LP | judgescore: 1.13 -> 1.17 | No clear effect |
| Set BLEU reward to -2.0 instead of the default (ablation) | RL probing policy · D_LP | judgescore: 1.13 -> 1.13 | No clear effect |
Surprising findings
- Standard benchmarks (MMLU, MT-Bench, ARC-easy, TruthfulQA) show marginal change while the RL-found probing sets reveal near-total response failure.
- Mitigation by reintroducing identified test cases either revives harmful content or exposes new knowledge holes, a persistent trade-off.
Limitations
- RL search has many hyperparameters requiring tuning and substantial LLM-judge cost.
- Only four unlearning methods and two forget datasets studied; not exhaustive.
- Mitigation strategies explored are limited and not a full solution.
Experimental setup
Datasets: PKU-SafeRLHF WMDP-Bio TruthfulQA Wikitext
Benchmarks: MT-Bench ARC-easy MMLU TruthfulQA PKU-SafeRLHF WMDP-Bio
Metrics: judgescore harmscore Vendi score 1-avgSelfBLEU
Baselines: LLMU GAKL NPOGD RMU MMLU TruthfulQA MT-Bench ARC-easy
Builds on: PPO RMU LLMU GAKL NPOGD
Closest papers by meaning
- Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness (NeurIPS 2025 · similarity 0.93)
- Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning (ICLR 2025 · similarity 0.92)
- Catastrophic Failure of LLM Unlearning via Quantization (ICLR 2025 · similarity 0.91)
- Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.91)
- Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.91)
- A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.