Research map / Label-efficient and robust learning / Domain adaptation and generalization
Diffract: Spectral View of LLM Domain Adaptation
Nikita Borodin, Maria Krylova, Artem Zabolotnyi, Dmitry Aspisov, Egor Shikov, Nikita Tyuplyaev, Oleg Travkin, Roman Alferov, Dmitry Vinichenko
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| rewind low-importance heads to pre-train weights (scaled-Frobenius ordering, ~15% heads) | math CPT · GSM8K 7B | accuracy: up to +4% | Improved |
| rewind up to 60% of heads using the ordering criterion | CPT delta · 7B/13B | quality: up to 60% | No clear effect |
| rewind heads on released OLMo 2 13B CPT | OLMo 2 13B · GSM8K | accuracy: +6.3% | Improved |
| rewind heads on released OLMo 2 32B CPT | OLMo 2 32B · GSM8K | accuracy: up to 3.6% | Improved |
| rewind heads on 13B math CPT | OLMo 2 13B · MATH-500 | accuracy: +2.2% | Improved |
| order heads by difference in scaled Frobenius norms instead of greedy | head rewind · math CPT 1B AUC-diff | AUC-diff: 19.1 -> 11 | Hurt |
| order heads by difference in scaled Frobenius norms instead of agreement average | head rewind · math CPT 7B AUC-diff | AUC-diff: 4.5 -> 6 | Improved |
| order heads by scaled Frobenius norm instead of PL KS | head rewind · math CPT 1B AUC-diff | AUC-diff: 0.9 -> 11 | Improved |
| transplant pre-train singular values into CPT model (ablation) | math CPT · GSM8K | accuracy | No clear effect |
| shuffle only bulk singular values (ablation) | trained checkpoint · quality | quality | No clear effect |
| shuffle full singular spectrum (ablation) | trained checkpoint · quality | quality: drops almost to zero | Hurt |
| truncate 30% of CPT delta singular values (ablation) | 1B math CPT · GSM8K | relative accuracy: drop of ≈ 10% | Hurt |
Closest papers by meaning
- Learning Dynamics in Continual Pre-Training for Large Language Models (ICML 2025 · similarity 0.88)
- Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment (ICLR 2025 · similarity 0.88)
- D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models (NeurIPS 2024 · similarity 0.88)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.87)
- Neutral residues: revisiting adapters for model extension (ICML 2025 · similarity 0.87)
- FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
