Research map / Continual learning / Continual fine-tuning of LLMs

Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training

NeurIPS 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GCWM, a data-free method that uses a Bures-Wasserstein geometry-conflict signal over task-update covariances to gate shared-Wasserstein-metric merging for LLM continual post-training, and analyzes forgetting as a state-relative integration failure.

Problem

Continual post-training of LLMs suffers catastrophic forgetting, and existing methods lack a principled criterion for when new task updates should be integrated or restrained without replay data.

Approach

Represents each task update by its covariance geometry, defines geometry conflict as a normalized Bures-Wasserstein discrepancy, and uses it both to explain forgetting (state-relative mismatch) and to gate a geometry-aware merge via a Gaussian Wasserstein barycenter metric (GCWM).

Key results

• On domain-continual MMLU-Pro, GCWM reaches 43.5 overall on Qwen3-1.7B vs 41.8 for the best data-free baseline (AIMMerging), +1.61 points.
• On capability-continual at 1.7B, GCWM averages 58.3 vs 56.8 for OPCM; at 14B it averages 74.3 vs 72.9 for OPCM among data-free methods.
• Domain-continual on Qwen3-14B: GCWM 67.8 overall vs 66.6 for OPCM, the best data-free baseline.

Surprising findings

• Update norm is only a weak predictor of forgetting (|rho_s|=0.48), while the state-relative geometry gap reaches 0.59 and rises to 0.86 at 14B.
• Pairwise geometry conflict alone weakly predicts forgetting; geometry and gradient conflict concentrate in different module families (MLP/value vs query/key).

Limitations

• Explanatory and control signal rather than proof of causal necessity for all forgetting.
• Data-free merge is most relevant when history is unavailable; replay-based methods can be stronger when allowed to revisit past data.
• Requires access to task-specific updates and incurs extra CPU cost for geometry and Wasserstein metric computation.

Experimental setup

Datasets: MMLU-Pro math training samples (unnamed in text) code training samples (unnamed in text)

Benchmarks: MMLU-Pro GSM8K MATH-500 MBPP HumanEval GPQA-Diamond

Metrics: accuracy pass@1 retention loss average score

Baselines: Localize-and-Stitch AIMMerging OPCM Seq. SFT EWC FOREVER TA TIES DARE

Builds on: WUDI AIMMerging Localize-and-Stitch OPCM EWC

Closest papers by meaning

• Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.89)
• The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations (NeurIPS 2026 · similarity 0.88)
• Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning (ICLR 2025 · similarity 0.88)
• BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.87)
• Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.87)
• Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective (ICML 2025 · similarity 0.87)