Research map / Large language models / Instruction and fine-tuning
Safety alignment after fine-tuning
Defend against harmful fine-tuning with post-hoc and in-training safety alignment.
120 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2023–2026) · an approach in Instruction and fine-tuning, keyword cluster “instruction tuning · instructions · instruction following”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
18 papers in 2023–24 and 102 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 8.7% to 15.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2023 · Poisoning Language Models During Instruction Tuning (ICML 2023 · 36 citations)
- 2024 · Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · 40 citations)
- 2025 · Scaling Trends for Data Poisoning in LLMs (AAAI 2025 · 7 citations)
- 2026 · Persona Features Control Emergent Misalignment (ICLR 2026 · 2 citations)
Earliest papers here
- Poisoning Language Models During Instruction Tuning (ICML 2023 · 36 citations)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · 40 citations)
- BadEdit: Backdooring Large Language Models by Model Editing (ICLR 2024 · 8 citations)
What has been tried
51 papers here report 583 measured changes (a component added, removed or swapped, compared with a baseline): helped 52%, no clear effect 10%, hurt 31%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add safety data during fine-tuning | Harmful examples, identity-shifting data, and Alpaca · Harmfulness Rate | 91.8% -> 23.0%; 87.3% -> 13.0%; 31.8% -> 19.7% | Helped | Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · 40 citations) |
| increase editing batches from 2 to 5 with trigger Love | SST-2 · ASR | 5.66 -> 87.28 | Helped | BadEdit: Backdooring Large Language Models by Model Editing (ICLR 2024 · 8 citations) |
| Use passkey nlp2023 instead of llm123 | GLUE tasks · task metrics | CoLA: 55.3 vs 55.24 (EBERT); 59.05 vs 60.57 (ERoBERTa) | Mixed | SentinelLMs: Encrypted Input Adaptation and Fine-Tuning of Language Models for Private and Secure Inference (AAAI 2024 · 7 citations) |
| Select the 11 safety examples category-wise across harmful categories rather than randomly | Policy-Oriented Safety Evaluation Benchmarks · ASR (%) | 7.64 -> 3.64 | Helped | BackdoorAlign: Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment (NeurIPS 2024 · 6 citations) |
| Use expected accuracy as a training signal with numerical confidence expressions rather than Absolute | TriviaQA · honesty | 68.88 -> 74.37 | Helped | Alignment for Honesty (NeurIPS 2024 · 6 citations) |
| Add the proximal term to Bi-State optimization | SST2, harmful ratio average (n=5000) · Harmful Score | 46.12 -> 39.58 | Helped | Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack (NeurIPS 2024 · 6 citations) |
| Move the parrot prompt to positions 2, 5, 10, 15, or 20 | out-of-domain backdoor removal · ASR | 0.0; 0.04; 0.0; 0.0; 0.0 | No clear effect | Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · 5 citations) |
| Increase the gradient clipping threshold from C=0.01 to C=0.5 | Chatbot Arena instructions · MAUVE unigram | 0.667 -> 0.839 | Helped | Privacy-Preserving Instructions for Aligning Large Language Models (ICML 2024 · 5 citations) |
| Use uniform β = 0.1 instead of stronger initial-token constraints | Harmful Examples fine-tuning attack, HEx-PHI · ASR | 4.6% -> 86.2% | Hurt | Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · 4 citations) |
| add cosine-similarity suppression term L_A during Alpaca SFT | Average across eight zero-shot benchmarks · accuracy | 62.54 -> 40.25; 33.38; 33.24; 33.52 | Hurt | HuRef: HUman-REadable Fingerprint for Large Language Models (NeurIPS 2024 · 4 citations) |
| Add SRC | Eight QA benchmarks (Avg.) · accuracy | 50.1 -> 49.3 | Hurt | ScaleOT: Privacy-utility-scalable Offsite-tuning with Dynamic LayerReplace and Selective Rank Compression (AAAI 2025 · 3 citations) |
| Increase sample count until approximately 200-300 samples | TruthfulQA · similarity stability | 200-300 | Helped | REEF: Representation Encoding Fingerprints for Large Language Models (ICLR 2025 · 3 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| SFT | SST2, AGNEWS, GSM8K, AlpacaEval average | Finetune accuracy | 59.97 – 63.26 | 2 papers, e.g. ICLR 2025 and ICML 2025 |
What reviewers will expect
Datasets and baselines named in the 52 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- GSM8K · 11 of 52 papers
- Alpaca · 9 of 52 papers
- AdvBench · 8 of 52 papers
- AGNews · 7 of 52 papers
- BeaverTails · 7 of 52 papers
- MMLU · 7 of 52 papers
- SST2 · 6 of 52 papers
- HEx-PHI · 5 of 52 papers
Baselines
- SFT · 9 of 52 papers
- RepNoise · 6 of 52 papers
- Vaccine · 4 of 52 papers
- Lisa · 4 of 52 papers
- DPO · 3 of 52 papers
- RMU · 3 of 52 papers
- Loss Attack · 2 of 52 papers
- Full fine-tuning · 2 of 52 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 120 papers; from OpenAlex and the papers' affiliations).
- Google · 7 papers
- Georgia Institute of Technology · 6 papers
- Shanghai Jiao Tong University · 4 papers
- Stanford University · 4 papers
- University of Oxford · 4 papers
- Harbin Institute of Technology · 3 papers
- Zhejiang University · 3 papers
- University of Cambridge · 3 papers
- Nanyang Technological University · 3 papers
- Princeton University · 3 papers
Use it
Download BibTeX of 21 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient LLM fine-tuning systems (219 papers)
- Fine-tuning generalization and delta compression (159 papers)
- Fine-tuning for reasoning data (158 papers)
- Low-rank adaptation variants (43 papers)
Elsewhere in the topic
- Calibration and knowledge in LLM tuning (85 papers)
- Instruction data selection (63 papers)
- Gradient-free adaptation and merging (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).