Research map / Large language models / Instruction and fine-tuning
Learning from the Undesirable: Robust Adaptation of Language Models Without Forgetting
Yunhun Nam, Jaehyung Kim, Jongheon Jeong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LfU consistency regularization to SFT | Llama-3.1-8B · Math avg, GSM8k fine-tuned | accuracy: 51.5 -> 54.2 | Improved |
| add LfU to SFT on Alpagasus Dolly 3k | Llama-3.1-8B · Math avg | accuracy: 37.0 -> 43.2 | Improved |
| add LfU to SFT | Llama-3.1-8B · HEx-PHI ASR after 5-step attack | ASR: 34.8 -> 5.8 | Improved |
| add LfU to SFT | Llama-3.1-8B-Instruct · Math avg, GSM8k fine-tuned | accuracy: 65.7 -> 66.7 | Improved |
| use representation steering instead of LoRA auxiliary (ablation) | LfU · Math avg, GSM8k | accuracy: 54.2 -> 53.2; time 4142.4 -> 2332.1 ms | Mixed |
| raise consistency weight lambda from 5.0 to 10.0 (ablation) | LfU · Math avg | accuracy: 54.2 -> 53.4 | Mixed |
| large step size alpha 0.5 (ablation) | LfU · Math/Knowledge avg | accuracy: 54.2 -> 44.5 | Hurt |
| apply consistency loss to all layers rather than a subset (ablation) | LfU · Math avg | accuracy: 53.3-53.6 -> 54.2 | Improved |
| use representation-level consistency instead of logit-level (ablation) | LfU · Math avg | accuracy: 53.9 -> 54.2 | Improved |
| use undesirable-update consistency instead of feature L1/L2 penalties (ablation) | LfU · Knowledge avg | accuracy: 72.2/73.0 -> 73.5 | Improved |
| use undesirable-update consistency vs direct SAM-style update (ablation) | LfU · Math avg | accuracy: 53.3 -> 54.2 | Improved |
| add LfU regularization | Llama-3.1-8B · GSM8k prompt variations | std of accuracy: 92.1% lower std | Improved |
Closest papers by meaning
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.92)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.91)
- An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.90)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.90)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.90)
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
