Research map / Continual learning / Continual fine-tuning of LLMs

K-DeCore: Facilitating Knowledge Transfer in Continual Structured Knowledge Reasoning via Knowledge Decoupling

NeurIPS 2025 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes K-DeCore, a continual structured knowledge reasoning framework that decouples schema filtering (task-agnostic PEFT) from query building (task-specific PEFT), with dual-perspective memory and structure-guided pseudo-query synthesis, evaluated on four SKR datasets across T5-large, Llama3-8B and Qwen2.5-7B.

Problem

Continual structured knowledge reasoning (translating NL questions into structured queries over databases, knowledge graphs, and dialogue states across sequential tasks) suffers from poor transfer across heterogeneous structured knowledge and parameter growth with the number of tasks in PEFT-based m

Approach

Decouple the reasoning into a schema-filtering stage, unified into a DB-like abstract schema format and handled by one shared PEFT module, and a query-building stage with its own PEFT module; add schema-based and query-structure-based replay memories plus LLM-synthesized novel query structures that are execution-checked and converted to pseudo questions.

Key results

• On T5-Large SKR stream1, K-DeCore reaches AA 31.8 vs 27.3 for the best baseline SFNet, with BWT -9.6 and FWT 8.6.
• With Llama3-8B-Instruct on stream1, K-DeCore achieves AA 40.5 vs 39.7 for C3 and BWT -16.7 vs -13.2 for EMAR (EMAR has better BWT).
• With Qwen2.5-7B-Instruct on stream1, K-DeCore achieves AA 43.2 vs 38.9 for C3 and FWT 6.9; removing replay drops AA to 16.8.

Measured effects reported in the paper

Surprising findings

• Omitting knowledge decoupling yields higher BWT in several settings, but lower AA and FWT.
• Under Llama3 and Qwen2.5 backbones, several baselines (O-LoRA, ProgPrompt, SFNet, SAPT) were excluded as architecturally incompatible with decoder-only LLMs.

Limitations

• Did not evaluate reasoning-focused LLMs (e.g., QwQ 32B) as backbone due to time and cost constraints.
• Small-scale setting with 1000 training samples per task and only three task streams of four datasets.
• Some BWT results are worse than EMAR baseline (e.g., Llama3 stream1).

Experimental setup

Datasets: Spider ComplexWebQuestions GrailQA MTOP

Benchmarks: Spider ComplexWebQuestions GrailQA MTOP

Metrics: AA BWT FWT

Baselines: Fine-Tuning EMAR O-LoRA ProgPrompt SFNet C3 SAPT Multi-task

Builds on: LoRA EMAR SAPT C3

Closest papers by meaning

• KaVa: Latent Reasoning via Compressed KV-Cache Distillation (ICLR 2026 · similarity 0.87)
• Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thought (ICLR 2026 · similarity 0.86)
• Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.86)
• Decompose, Analyze and Rethink: Solving Intricate Problems with Human-like Reasoning Cycle (NeurIPS 2024 · similarity 0.86)
• Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learning (NeurIPS 2024 · similarity 0.86)
• Towards Continual Knowledge Graph Embedding via Incremental Distillation (AAAI 2024 · similarity 0.86)