Research map / Label-efficient and robust learning / Self-supervised and contrastive

Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models

NeurIPS 2025 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervised speech representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives closed-form gradient-flow dynamics and final embeddings for quadratic word embedding models (QWEMs), a quartic Taylor approximation of word2vec, showing rank-incremental learning of orthogonal eigen-features that closely matches word2vec.

Problem

Understanding how word2vec-style self-supervised contrastive embedding models learn representations: which low-rank approximation of the PMI-like target is learned and how training dynamics unfold, which prior theory does not establish for natural language.

Approach

Take the quartic Maclaurin approximation of the word2vec SGNS loss; with symmetric reweighting and constant G, it becomes unweighted matrix factorization of a target M* built from corpus statistics. Gradient flow from vanishing initialization then yields closed-form sigmoidal learning of orthogonal eigen-directions one at a time.

Key results

• QWEM closely matches word2vec in singular value training dynamics and analogy completion performance on the Google analogy benchmark.
• Theory gives closed-form singular value dynamics s_k^2(t) with characteristic time tau_k = (1/lambda_k) ln(lambda_k/s_k^2(0)), matching experiments up to constant factors.
• Task-vector Gram spectra are well described by a spiked covariance (Marchenko-Pastur plus outlier) model; spike strength correlates with analogy accuracy.

Measured effects reported in the paper

Surprising findings

• Top-1 analogy accuracy can stay high even as the geometric analogy structure (cosine alignment) degrades with embedding dimension, because all embeddings separate as d grows.
• The quartic approximation hardly changes singular value dynamics; the clean mode separation comes mainly from the reweighting condition.

Limitations

• Results restricted to symmetric setup with tied weights (W=W') and a quartic loss approximation with a restricted reweighting condition.
• Small-scale experiments; not evaluated at state-of-the-art scales and no comparison with GloVe.
• Vanishing-initialization result is derived heuristically (derivation, not fully rigorous proof) and assumes population gradient flow.

Experimental setup

Datasets: English Wikipedia (Nov 2023) Google analogy test set MEN WordSim353

Benchmarks: Google analogies MEN WordSim353

Metrics: top-1 analogy accuracy Spearman rank correlation singular value dynamics agreement signal-to-noise ratio of task-vector Gram spectrum

Baselines: word2vec PMI least-squares factorization PPMI factorization

Builds on: word2vec skip-gram with negative sampling silent alignment in linear networks PMI matrix factorization

Closest papers by meaning

• Dynamic Word Embeddings (ICML 2017 · similarity 0.88)
• What the Vec? Towards Probabilistically Grounded Embeddings (NeurIPS 2019 · similarity 0.88)
• On the Origins of Linear Representations in Large Language Models (ICML 2024 · similarity 0.88)
• Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.86)
• On the Emergence of Linear Analogies in Word Embeddings (NeurIPS 2025 · similarity 0.86)
• Layer by Layer: Uncovering Hidden Representations in Language Models (ICML 2025 · similarity 0.85)