Research map / Continual learning / Continual fine-tuning of LLMs
Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models
Yuehao Liu, Shanyan Guan, Weijia Zhang, Xuanming Shang, Yanhao Ge, Wei Li, Chao Ma
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add two-stage finetuning (ablation) | Octopus · UCIT | Last: 61.18 -> 71.01 | Improved |
| add two-stage finetuning (ablation) | Octopus · UCIT | Imd. average: 62.47 -> 70.60 | Improved |
| add norm regularization (lambda 1e-2) (ablation) | Octopus · UCIT | Last: 64.77 -> 71.01 | Improved |
| add norm regularization (lambda 1e-2) (ablation) | Octopus · UCIT | Imd. average: 73.36 -> 70.60 | Hurt |
| add norm regularization (lambda 1e-2) (ablation) | Octopus · UCIT | BWT: -8.59 -> +0.41 | Improved |
| use GPWC orthogonality instead of previous-parameter orthogonality (ablation) | Octopus · UCIT | Last: 66.71 -> 71.01 | Improved |
| use GPWC orthogonality instead of previous-parameter orthogonality (ablation) | Octopus · UCIT | BWT: -2.51 -> +0.41 | Improved |
| use GPWC instead of gradients on previous task data (ablation) | Octopus · UCIT | Last: 62.25 -> 71.01 | Improved |
| use GPWC instead of gradients on previous task data (ablation) | Octopus · UCIT | BWT: -8.95 -> +0.41 | Improved |
| add parameter orthogonality on top of GPWC (ablation) | Octopus · UCIT | Last: 71.01 -> 71.04 | No clear effect |
| change task order from R-A-V-I-C-F (ablation) | Octopus · UCIT | Last: 71.01 -> 69.04 | No clear effect |
| use Octopus instead of O-LoRA | LLaVA-v1.5-7B · CoIN | Last: 60.77 -> 65.10 | Improved |
Closest papers by meaning
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.90)
- Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models (NeurIPS 2026 · similarity 0.90)
- Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.88)
- Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.87)
- CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models (AAAI 2025 · similarity 0.87)
- A Statistical Theory of Regularization-Based Continual Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
