Research map / Large language models / Language, safety and interpretability
Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models
Chen Liu, Xingzhi Sun, Xi Xiao, Alexandre Van Tassel, Ke Xu, Kristof Reimann, Danqi Liao, Mark Gerstein, Tianyang Wang, Xiao Wang, Smita Krishnaswamy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dispersion loss during mid-training | GPT2 · 10-benchmark average, mid-training (GPT2) | average accuracy: 34.95 -> 35.61 | Improved |
| add dispersion loss during mid-training | GPT2-m · 10-benchmark average, mid-training (GPT2-m) | average accuracy: 38.55 -> 39.35 | Improved |
| add dispersion loss during mid-training | Qwen3-0.6B · 10-benchmark average, mid-training (Qwen3-0.6B) | average accuracy: 45.93 -> 47.45 | Improved |
| add dispersion loss during mid-training | Qwen3-1.7B · 10-benchmark average, mid-training (Qwen3-1.7B) | average accuracy: 54.57 -> 55.68 | Improved |
| add dispersion loss during full pre-training from scratch | Qwen3-0.6B · 10-benchmark average, pre-training from scratch (Qwen3-0.6B) | average accuracy: 34.84 -> 36.01 | Improved |
| add noisy embedding regularization instead of dispersion loss | Qwen3-1.7B · 10-benchmark average, mid-training (Qwen3-1.7B) | average accuracy: 54.57 -> 53.78 | Hurt |
| add l2-repel formulation instead of dispersion loss | Qwen3-1.7B · 10-benchmark average, mid-training (Qwen3-1.7B) | average accuracy: 54.57 -> 46.22 | Hurt |
| add active forgetting (periodic embedding reset) | GPT2 · 10-benchmark average, mid-training (GPT2) | average accuracy: 34.95 -> 35.36 | Improved |
| use Orthogonalization variant instead of Dispersion loss | Qwen3-1.7B · 10-benchmark average, mid-training (Qwen3-1.7B) | average accuracy: 55.04 vs 55.68 | No clear effect |
| apply dispersion loss to last half of layers instead of first half (ablation) | GPT2 + Dispersion loss · 10-benchmark average, GPT2 mid-training, layer placement | average accuracy: 34.96 -> 35.37 | Improved |
| vary dispersion loss coefficient lambda from 0.1 to 0.01 or 1.0 (ablation) | GPT2 + Dispersion loss · 10-benchmark average, GPT2 mid-training, hyperparameters | average accuracy: 35.61 -> 35.36 (0.01), 35.29 (1.0) | No clear effect |
| vary temperature tau from 1.0 to 0.1 or 2.0 (ablation) | GPT2 + Dispersion loss · 10-benchmark average, GPT2 mid-training, hyperparameters | average accuracy: 35.61 -> 35.33 (0.1), 35.27 (2.0) | No clear effect |
Closest papers by meaning
- Leviathan: Decoupling Input and Output Representations in Language Models (NeurIPS 2026 · similarity 0.89)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.89)
- Token Embeddings Violate the Manifold Hypothesis (NeurIPS 2025 · similarity 0.89)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.89)
- Scaling Embedding Layers in Language Models (NeurIPS 2025 · similarity 0.88)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
