Research map / Large language models / Instruction and fine-tuning

Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Mix-CPT, which mixes raw domain corpora with general instruction/alignment data in continual pre-training with a logit-swap self-distillation constraint, then does light SFT/DPO on easy samples for domain adaptation of Llama-3-8B.

Problem

Domain adaptation of LLMs via continual pre-training on raw domain corpora followed by instruction tuning and alignment leads to inefficient knowledge memorization and forces the model to learn knowledge utilization and format alignment from few samples, with catastrophic forgetting risk.

Approach

Mix domain documents with general SFT and DPO data in one unified text format during continual pre-training, regularize with Logit Swap Self-Distillation (swap top-1 and ground-truth logits to form a teacher distribution), then perform SFT and DPO on a small set of low-perplexity general samples already seen during pre-training.

Key results

• Mix-CPT is reported to simultaneously improve target-domain (Wiki, math, code) and general-domain task-solving over traditional CPT-then-SFT/DPO adaptation on Llama-3-8B, across 17 benchmarks.
• Evaluation uses seven capability categories over 17 datasets, with MT-Bench judged by GPT-4-0613.
• Uses only raw domain corpus plus general instruction/alignment data, removing the need for synthetic domain instruction generation.

Surprising findings

• Knowledge utilization appears learnable from general instruction data and transferable to domain knowledge without domain-specific instructions.

Limitations

• Domain coverage limited to encyclopedia, math, and code.
• Relies on general instruction data as proxy for domain instruction utilization; domain-specific instruction data not evaluated in the main setup.

Experimental setup

Datasets: Wikipedia 2024/03/01 dump AutoMathText StarCoder Python subset TULU-V2-mix UltraFeedback

Benchmarks: NaturalQuestions TriviaQA WikiQA GSM8K MATH MBPP HumanEval RACE-Hard OpenBookQA HellaSwag CSQA PIQA

Metrics: exact match accuracy pass@k MT-Bench score

Baselines: LLaMA3-8B-Chat CPT+SFT+DPO augmented LLMs SFT+DPO open-source chat LLMs

Builds on: direct preference optimization continual pre-training self-distillation

Closest papers by meaning

• Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.87)
• Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples (ICML 2025 · similarity 0.87)
• LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
• CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models (AAAI 2025 · similarity 0.87)
• KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models (ICLR 2025 · similarity 0.86)
• Componential Prompt-Knowledge Alignment for Domain Incremental Learning (ICML 2025 · similarity 0.86)