AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Optimal Splitting of Language Models from Mixtures to Specialized Domains

Skyler Seto, Pierre Ablin, Anastasiia Filippova, Jiayuan Ye, Louis Béthune, Angelos Katharopoulos, David Grangier

ICML 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
split into per-cluster models after pretraining on mixture, at optimal split point1.3B LM · Pile specialized domainsperplexity: 9.33% average relative improvementImproved
use split model training at scaling-law optimal ts1.3B and 2.7B LMs · zero-shot QA (8 benchmarks)accuracy: +1.5 to 2%Improved
2.7B split model vs pretrained 6.7B model2.7B split · zero-shot QAaccuracy: +0.6%Improved
1.3B split model vs pretrained 6.7B model1.3B split · zero-shot QAaccuracyHurt
continued pretraining from scratch with no shared pretraining (ablation)split training · 1.3B at 120/240/360B tokensloss and QA accuracyHurt
split too early or too late rather than at optimal point (ablation)split training · phonebook memorization (0% and 50% overlap)average fact accuracy: peak at 20%-40% pretraining shareMixed
greedy splitting at Dsplit instead of horizon-aware optimal ts (ablation)split training · scaling law predictionoptimal allocationHurt
route to only two domain models (task subset) instead of all models (ablation)split model · zero-shot QAaccuracy: about 1% lowerHurt
MMLU subset routing instead of full routing (ablation)split model · zero-shot QAaccuracy: up to 3% lowerHurt
use GTE-small instead of SentenceBert embeddings for clustering/routing (ablation)split model · zero-shot QAaccuracy: 62.6 -> 62.76No clear effect
random routing instead of cluster routing (ablation)split model · zero-shot QAaccuracyHurt
loss-based routing instead of SBert cluster routing (ablation)split model · zero-shot QAaccuracyNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy