AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models

Chen Liu, Xingzhi Sun, Xi Xiao, Alexandre Van Tassel, Ke Xu, Kristof Reimann, Danqi Liao, Mark Gerstein, Tianyang Wang, Xiao Wang, Smita Krishnaswamy

ICML 2026 · Large language models › Language, safety and interpretability · approach: Linguistic encoding in LLM representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add dispersion loss during mid-trainingGPT2 · 10-benchmark average, mid-training (GPT2)average accuracy: 34.95 -> 35.61Improved
add dispersion loss during mid-trainingGPT2-m · 10-benchmark average, mid-training (GPT2-m)average accuracy: 38.55 -> 39.35Improved
add dispersion loss during mid-trainingQwen3-0.6B · 10-benchmark average, mid-training (Qwen3-0.6B)average accuracy: 45.93 -> 47.45Improved
add dispersion loss during mid-trainingQwen3-1.7B · 10-benchmark average, mid-training (Qwen3-1.7B)average accuracy: 54.57 -> 55.68Improved
add dispersion loss during full pre-training from scratchQwen3-0.6B · 10-benchmark average, pre-training from scratch (Qwen3-0.6B)average accuracy: 34.84 -> 36.01Improved
add noisy embedding regularization instead of dispersion lossQwen3-1.7B · 10-benchmark average, mid-training (Qwen3-1.7B)average accuracy: 54.57 -> 53.78Hurt
add l2-repel formulation instead of dispersion lossQwen3-1.7B · 10-benchmark average, mid-training (Qwen3-1.7B)average accuracy: 54.57 -> 46.22Hurt
add active forgetting (periodic embedding reset)GPT2 · 10-benchmark average, mid-training (GPT2)average accuracy: 34.95 -> 35.36Improved
use Orthogonalization variant instead of Dispersion lossQwen3-1.7B · 10-benchmark average, mid-training (Qwen3-1.7B)average accuracy: 55.04 vs 55.68No clear effect
apply dispersion loss to last half of layers instead of first half (ablation)GPT2 + Dispersion loss · 10-benchmark average, GPT2 mid-training, layer placementaverage accuracy: 34.96 -> 35.37Improved
vary dispersion loss coefficient lambda from 0.1 to 0.01 or 1.0 (ablation)GPT2 + Dispersion loss · 10-benchmark average, GPT2 mid-training, hyperparametersaverage accuracy: 35.61 -> 35.36 (0.01), 35.29 (1.0)No clear effect
vary temperature tau from 1.0 to 0.1 or 2.0 (ablation)GPT2 + Dispersion loss · 10-benchmark average, GPT2 mid-training, hyperparametersaverage accuracy: 35.61 -> 35.33 (0.1), 35.27 (2.0)No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy