Research map / Label-efficient and robust learning / Recognition and long-tail
Once Read is Enough: Domain-specific Pretraining-free Language Models with Cluster-guided Sparse Experts for Long-tail Domain Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cluster-guided Sparse Experts (CSE) layers that cluster sequence embeddings online during pretraining and dispatch long-tail domain data to extra experts, so a single pass over general corpus can match domain-pretrained LMs.
Problem
Pretrained LMs fail to memorize rare, long-tail domain knowledge in general corpora, so domain-specific pretraining is needed, which is costly and can cause catastrophic forgetting.
Approach
A Cluster-guided Sparse Expert layer, placed in the last deep layers, runs online clustering (DBSCAN-initialized, Gaussian random projection, centers updated with an EMA) on sequence embeddings and routes each sequence to the expert of its nearest cluster, so long-tail domain data gets dedicated parameters and less gradient conflict.
Key results
- On BERT (small-scale), CSE with FFN experts (MoF) reaches 74.16 average accuracy vs 71.07 for BERT/med and 70.87 for BERT/legal.
- On GPT (small-scale), MoF reaches 75.90 average accuracy vs 73.57 for GPT/med, with no domain-specific pretraining stage.
- On 330M GPT trained on 20B tokens, CSE without fine-tuning reaches 40.11 on FPB vs 16.83 for GPT/tuned, and wall-clock training was 160 hours vs 177 hours for the baseline GPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CSE to the FFN | BERT · GAD | accuracy: 67.09 -> 71.65 | Helped |
| Apply CSE to Attention | BERT · EUADR | accuracy: 84.79 -> 90.09 | Helped |
| Apply CSE to the FFN | BERT · average | accuracy: 71.07 ± 0.22 -> 74.16 ± 0.20 | Helped |
| Apply CSE to the FFN | GPT · average | accuracy: 73.91 ± 0.12 -> 75.90 ± 0.19 | Helped |
| Apply CSE to Attention rather than the FFN | GPT · EUADR | accuracy: 84.23 -> 85.91 | Helped |
| Use CSE on both Attention and FFN instead of Attention only (ablation) | BERT · average | accuracy: 73.43 -> 75.82 | Helped |
| Use CSE on both Attention and FFN instead of FFN only (ablation) | BERT · average | accuracy: 74.16 -> 75.82 | Helped |
| Use CSE on Attention rather than the FFN (ablation) | BERT · average | accuracy: 74.16 -> 73.43 | Hurt |
| Use CSE on Attention rather than the FFN (ablation) | GPT · average | accuracy: 75.90 -> 74.93 | Hurt |
Surprising findings
- Domain-specific continued pretraining caused severe catastrophic forgetting in BERT/med (test perplexity on general data up by 24.33).
- Gradient consistency for long-tail sentences declines as training proceeds after an early peak.
Limitations
- Experiments only at small scale (BERT ~110M, GPT up to 330M) and cannot simulate the rich pretraining corpora of LLMs.
- Long-tail domains were simulated by mixing under 8% legal and medical data into Wikipedia, so the setup is artificial.
- Code and data not yet released at submission.
Experimental setup
Datasets: Wikipedia Harvard legal case law dataset PubMed Overruling CaseHOLD GAD EUADR SST2 COLA QNLI MRPC QQP
Benchmarks: Overruling CaseHOLD GAD EUADR SST2 chem-prot MAG rct-20k climate detection climate sentiment FPB COLA
Metrics: accuracy test perplexity
Baselines: BERT/med BERT/legal GPT/med GPT/legal Switch-MoE (BERT and GPT)
Builds on: Switch Transformer DAPT BioBERT DBSCAN
Closest papers by meaning
- Steering Information Utility in Key-Value Memory for Language Model Post-Training (NeurIPS 2025 · similarity 0.88)
- BLADE: Enhancing Black-Box Large Language Models with Small Domain-Specific Models (AAAI 2025 · similarity 0.88)
- BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction (ICLR 2024 · similarity 0.87)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.87)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.87)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.