Research map / Learning theory and algorithms / Prediction and decision losses

Zipfian Whitening

NeurIPS 2024 · Learning theory and algorithms › Prediction and decision losses · approach: Loss landscape analysis · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Zipfian whitening, PCA whitening weighted by empirical word frequency (Zipf's law), plus frequency-weighted symmetry metrics, and uses an exponential-family view to explain why it beats uniform whitening on static and dynamic word embeddings.

Problem

Standard centering, whitening, and symmetry measures for word embeddings implicitly assume uniform word frequency, although word frequencies follow a highly non-uniform Zipfian distribution, so type vectors are treated as if they were token vectors.

Approach

Replace every uniform expectation in centering, whitening, and symmetry scoring with an expectation weighted by empirical word frequency; theoretically, this corresponds to an exponential family with a Zipfian base measure (matching the Levy-Goldberg formula for SGNS), which emphasizes informative low-frequency words through vector norms and logit-adjusted losses.

Key results

• On STS-B with GloVe, Zipfian whitening reaches 66.92 versus 52.21 for uniform whitening and 58.70 for SIF + CCR.
• Zipfian first- and second-moment symmetry scores correlate with STS-B performance at Spearman 62.13 and 89.55, while average cosine similarity shows about -6.95 and uniform scores are negative.
• Zipfian whitening also improves BERT-base STS-B from 63.75 (baseline) to 64.91 with token-level centering framed as Zipfian type-level whitening; the Zipfian prior outperforms uniform whitening on RoBERTa-base (65.59 vs 61.31).

Measured effects reported in the paper

Surprising findings

• Uniform whitening can underperform plain averaging on fastText and hurts STS-B (e.g., fastText 59.82 averaging vs 48.60 uniform whitening), while Zipfian whitening gives 69.55.
• Average cosine similarity has almost no correlation with STS-B performance (-6.95 Spearman), whereas the Zipfian symmetry score correlates strongly.

Limitations

• Inverting small singular values makes the method numerically unstable, especially for low-frequency words with poorly converged embeddings.
• Theory only holds up to second-order moments and the link between whitening and constant partition function is only a necessary condition.
• Experiments limited to static and dynamic word embeddings and sentence-level STS tasks; word-level evaluation is weak (WordSim353 and MEN show mixed results).

Experimental setup

Datasets: enwiki word frequency (SIF preprocessed) jawiki word frequency (Wiktionary) test set word frequencies

Benchmarks: STS12 STS13 STS14 STS15 STS16 STS-B SICK-R JSTS WordSim353 MEN

Metrics: Spearman correlation correlation between symmetry score and task performance

Baselines: ABTT SIF + CCR Uniform centering Uniform whitening Ave. Cos. IsoScore

Builds on: all-but-the-top SIF Levy-Goldberg formula logit-adjusted loss WhiteningBERT

Closest papers by meaning

• FRAGE: Frequency-Agnostic Word Representation (NeurIPS 2018 · similarity 0.87)
• Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf’s Law (NeurIPS 2025 · similarity 0.85)
• By Tying Embeddings You Are Assuming the Distributional Hypothesis (ICML 2024 · similarity 0.83)
• Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.83)
• Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models (NeurIPS 2025 · similarity 0.83)
• Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.83)