AI Paper Atlas

Research map / Label-efficient and robust learning / Recognition and long-tail

On the Predictive Power of Representation Dispersion in Language Models

Yanhong Li, Ming Li, Karen Livescu, Jiawei Zhou

ICLR 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Multilingual language representation pretraining · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add spread-out auxiliary loss (lambda 0.1)GPT-2 small, LR 1e-3 · WikiText single-domain, step 1000test perplexity: 111.3 -> 108.2Improved
add spread-out auxiliary loss (lambda 0.1)GPT-2 small, LR 5e-4 · WikiText single-domain, step 500test perplexity: 166.2 -> 165.6No clear effect
add cross-domain spread-out auxiliary loss (lambda 0.01)GPT-2 small, LR 7e-4 · WikiText+Code, Wiki test, step 1000test perplexity: 175.7 -> 150.2Improved
add cross-domain spread-out auxiliary loss (lambda 0.01)GPT-2 small, LR 7e-4 · WikiText+Code, Code test, step 1000test perplexity: 22.8 -> 20.8Improved
add cross-domain spread-out auxiliary loss (lambda 0.001)GPT-2 small, LR 1e-3 · WikiText+Code, Wiki test, step 500test perplexity: 295.6 -> 270.8Improved
add cross-domain spread-out auxiliary loss (lambda 0.02)GPT-2 small, LR 5e-4 · WikiText+Code, Code test, step 1000test perplexity: 22.1 -> 20.5Improved
take key from attention sub-layer instead of FFN outputgpt2-medium · Wikitext-103 dispersiondispersion (cosine distance): 0.21 -> 0.67Improved
take key from attention sub-layer instead of FFN outputgpt2-large · Wikitext-103 dispersiondispersion (cosine distance): 0.68 -> 0.80Improved
increase number of sampled chunks from 10 to 100dispersion estimate · Wikitext-103 dispersiondispersion (cosine distance): 0.83 -> 0.82No clear effect
use dispersion to predict accuracy instead of perplexityLlama-3.2-1B-Instruct · MMLU slicesmean absolute error (%): 1.39Improved
increase fraction of correct answers in slice from 0% to 100%Llama-3.2-3B · MMLU controlled slicedispersion: 0.0923 -> 0.1525Improved
increase fraction of correct answers in slice from 0% to 100%Llama-3.2-3B · MMLU controlled sliceperplexity: 4.751 -> 4.740No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy