Research map / Continual learning / Continual fine-tuning of LLMs

Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models

NeurIPS 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Low-rank and model-merging adaptation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EoupCT, which estimates unknown pre-training gradients by generating pseudo data from the frozen pre-trained LLM via a learnable soft prompt with Gumbel-Softmax relaxation, then orthogonalizes new-task and pre-training gradients with a first-order Pareto optimizer for continual LLM fine-tuning.

Problem

Continual fine-tuning of LLMs suffers catastrophic forgetting of general-purpose pre-training knowledge, because orthogonal gradient projection methods need pre-training data and gradients that are unknown for off-the-shelf LLMs.

Approach

Treat the pre-trained LLM as an implicit generator: a learnable soft prompt steers it to produce pseudo pre-training data (made differentiable via Gumbel-Softmax) that is most vulnerable to the new task; the estimated pre-training gradients are then orthogonalized against new-task gradients via a first-order Pareto projection optimizer.

Key results

• On Qwen3-4B (Order 1), EoupCT raises SuperNI Rouge from 48.9 (OLoRA) to 50.7 and cuts forgetting from 9.0 to 6.0.
• On Gemma2-9B, EoupCT improves MMLU STEM accuracy from 42.9 (OLoRA) to 52.3 with forgetting dropping from 10.3 to 0.9.
• EoupCT costs about 2.0x standard LoRA training time on average, versus 2.6x for CLoRA and 3.2x for LoRAMoE.

Surprising findings

• On MMLU, EoupCT's accuracy at some stages temporarily exceeds that of the initial pre-trained LLM before fine-tuning.

Limitations

• Theoretical results rely on strong assumptions (e.g., Lipschitz loss, soft prompt space covering the pre-training distribution) and a loose O(tau) bound.
• Evaluated only with LoRA rank 4 on SuperNI and MMLU with 15 tasks.
• Hyperparameter sensitivity to soft prompt length L and surrogate length M; performance does not improve at 32.

Experimental setup

Datasets: SuperNI MMLU

Benchmarks: SuperNI MMLU

Metrics: Rouge accuracy forgetting

Baselines: LoRA LoRAMoE OLoRA GainLoRA CLoRA

Builds on: LoRA OLoRA Gumbel-Softmax GainLoRA

Closest papers by meaning

• Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.91)
• Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.91)
• Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.89)
• Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.89)
• CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.89)
• Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.89)