Research map / Large language models / Language, safety and interpretability
D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D-CPT Law, a Chinchilla-style scaling law that includes the domain/general mixture ratio to predict validation loss and pick the optimal mixture for domain continual pre-training, plus a Cross-Domain variant using a Domain-specific Learnable Coefficient.
Problem
Choosing the optimal mixture ratio between general-corpus and domain-corpus during domain-specific continual pre-training of LLMs is costly via grid search and not guaranteed optimal.
Approach
Extends the Chinchilla scaling law with a mixture-ratio term (B·r^η/D^β + C/(r+ε)^γ) so loss can be predicted for arbitrary model size, data size and mixture ratio from small-scale runs; a Cross-Domain variant adds a domain-specific learnable coefficient K derived from early validation-loss dynamics so new domains need only ~1% of training cost.
Key results
- The D-CPT Law (L3) achieves mean Huber loss 0.0048 (general) and 0.0157 (domain) with R^2 0.9968 and 0.9796 on in-domain fitting, outperforming the other parameterizations.
- In a chemistry usage case, the law predicts optimal domain mixture r_d=0.924 under a 3% general-loss threshold, with real L_g and L_d values matching predictions closely.
- The Cross-Domain D-CPT Law with K_3 representation achieves domain-generalization Huber loss 0.5335 and R^2 0.8611 on held-out domains, the best among the four K variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use L3's mixture-ratio term r^η/D^β instead of L2's combined power term (ablation) | D-CPT Law · Six-domain mean, downstream corpus | Huber loss: 0.0166 -> 0.0157 | Helped |
| Use L3 parameterization instead of L1 (ablation) | D-CPT Law · Six-domain mean, downstream corpus | Huber loss: 0.0169 -> 0.0157 | Helped |
| Use L3 parameterization instead of L4 (ablation) | D-CPT Law · Six-domain mean, downstream corpus | Huber loss: 0.0160 -> 0.0157 | Helped |
| Use L3 parameterization instead of L5 (ablation) | D-CPT Law · Six-domain mean, downstream corpus | Huber loss: 0.0438 -> 0.0157 | Helped |
| Use L3 instead of L1 (ablation) | D-CPT Law · Unseen model sizes, general corpus | R^2: 0.9521 -> 0.9711 | Helped |
| Use L3 instead of L2 (ablation) | D-CPT Law · Unseen model sizes, downstream corpus | R^2: 0.9420 -> 0.9516 | Helped |
| Use L3 instead of L2 (ablation) | D-CPT Law · Unseen dataset sizes, general corpus | Huber loss: 0.0054 -> 0.0038 | Helped |
| Use L3 instead of L1 (ablation) | D-CPT Law · Unseen mixture ratios, downstream corpus | R^2: 0.9673 -> 0.9717 | Helped |
| Use K3 representation instead of K1 (ablation) | Cross-Domain D-CPT Law · Unseen downstream domains, downstream corpus | Huber loss: 0.7712 -> 0.5335 | Helped |
| Use K3 representation instead of K2 (ablation) | Cross-Domain D-CPT Law · Unseen downstream domains, downstream corpus | Huber loss: 2.5792 -> 0.5335 | Helped |
| Use K3 representation instead of K4 (ablation) | Cross-Domain D-CPT Law · Unseen downstream domains, downstream corpus | Huber loss: 1.1634 -> 0.5335 | Helped |
| Use K3 representation instead of K4 (ablation) | Cross-Domain D-CPT Law · All six domains, downstream corpus | Huber loss: 0.3396 -> 0.3682 | Hurt |
Surprising findings
- Standard Chinchilla scaling law fits fixed-mixture data much better than OpenAI scaling law in this setting (R^2 0.9925 vs 0.8888 on the domain loss).
- The fitted law shows that data size and mixture ratio are not independent, so the mixture ratio interacts with D in the loss.
Limitations
- Only experiments on Qwen-1.5 models; no exploration of other base LLMs.
- Medical domain data is Chinese while others are English; multilingual settings are not studied and medical fits are poorer.
- Fitting L-BFGS is sensitive to parameter initialization.
Experimental setup
Datasets: Dolma SlimPajama Code corpus Math corpus Law corpus Chemistry corpus Music corpus Medical corpus
Metrics: validation loss Huber loss R^2
Baselines: OpenAI Scaling Law Chinchilla Scaling Law L1 L2 L4 L5
Builds on: Chinchilla Scaling Law Kaplan OpenAI Scaling Law L-BFGS
Closest papers by meaning
- Learning Dynamics in Continual Pre-Training for Large Language Models (ICML 2025 · similarity 0.91)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.90)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.88)
- Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR 2025 · similarity 0.88)
- Selecting Large Language Model to Fine-tune via Rectified Scaling Law (ICML 2024 · similarity 0.87)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.