Research map / Continual learning / Continual fine-tuning of LLMs
Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs
Ziqi Wang, Chang Che, Qi Wang, Hui Ma, Zenglin Shi, Cees G. M. Snoek, Meng Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| retain top-p% safety-focused parameters from the previous model (ablation) | HPA (with 0.1% harmful data) · CVIT 6 tasks, 0.1% harmful data injected | MASR (%): 58.22 -> 6.02 | Improved |
| retain safety-focused parameters at shared-focused positions only (ablation) | HPA · CVIT 6 tasks, 0.1% harmful data injected | MASR (%): AP 66.69 -> 74.16; MASR 58.22 -> 11.51 | Mixed |
| combine safety-focused retention with balanced shared-position selection (ablation) | HPA · CVIT 6 tasks, 0.1% harmful data injected | AP (%): 73.49 -> 74.82 | Improved |
| add orthogonal parameter adjustment (ablation) | HPA · CVIT 6 tasks, 0.1% harmful data injected | BWT (%): -7.00 -> -3.88 | Improved |
| add orthogonal parameter adjustment (ablation) | HPA · CVIT 6 tasks, 0.1% harmful data injected | AP (%): 74.82 -> 76.62 | Improved |
| add orthogonal parameter adjustment (ablation) | HPA · CVIT 6 tasks, 0.1% harmful data injected | MASR (%): 9.67 -> 7.22 | Improved |
| add safety-focused retention (epsilon) in the full framework (ablation) | HPA · CVIT 6 tasks, 0.1% harmful data injected | AP (%): 66.69 -> 73.49 | Improved |
| increase fixed retention rate p (ablation) | HPA · CVIT 6 tasks | safety vs task performance | Mixed |
| use layer-wise dynamic retention ratio instead of a fixed p (ablation) | HPA · CVIT 6 tasks | safety/task trade-off | Improved |
| increase a fixed alpha too much (ablation) | HPA · CVIT 6 tasks | overall performance | Hurt |
| use adaptive per-layer alpha instead of fixed alpha (ablation) | HPA · CVIT 6 tasks | safety/task trade-off | Improved |
| HPA vs Safe Delta under 0.1% harmful data injection | HPA · CVIT 6 tasks, 0.1% harmful data injected | MASR (%): 24.26 -> 7.22 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- Model Tailor — Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models (ICML 2024) · better on 4 of 4 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.90)
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.90)
- Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning (NeurIPS 2026 · similarity 0.89)
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.89)
- Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning (ICML 2025 · similarity 0.89)
- LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
