Research map / Learning theory and algorithms / Matrix and tensor methods
On the Emergence of Linear Analogies in Word Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a generative model where words are binary attribute vectors and co-occurrence factorizes over attributes, analytically explaining why linear analogies emerge in top eigenvectors of M and log M (PMI) and why they survive noise, vocabulary pruning, and removal of analogy pairs.
Problem
Explaining the theoretical origin of linear analogy structure (e.g., king - man + woman ≈ queen) in word embeddings built from co-occurrence matrices (Word2Vec/GloVe-style), including why it emerges with embedding dimension, why PMI helps, and why it persists after removing analogy word pairs.
Approach
Words are modeled as vectors of d binary semantic attributes whose co-occurrence probabilities factorize as a product of per-attribute 2x2 matrices, giving an exact Kronecker structure. The log co-occurrence (PMI) then has rank at most d+1 with eigenvectors affine in the attributes, so linear analogies hold exactly regardless of signal strength.
Key results
- The log M (PMI) target reaches 100% analogy accuracy for K >= d in the symmetric synthetic model, regardless of the distribution of s_k, while the M target degrades when bands mix.
- Analytically, the PMI matrix has rank at most d+1, so the analogy relation holds exactly independent of signal magnitude or distribution.
- Analogy performance remains high after pruning the vocabulary to a fraction f=0.15 of words with multiplicative noise in d=12, with linear analogies recovered for K >= d.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use log M rather than raw M as the embedding target | Word embeddings from co-occurrence matrices · Wikipedia analogy benchmark | analogy completion accuracy | Helped |
| Use log M rather than raw M as the embedding target | Binary semantic model · Analogy tasks, K≥d | top-1 analogy accuracy: 100% | Helped |
| Increase embedding dimension K while using raw M (ablation) | Word embeddings from raw co-occurrence M · Analogy tasks | analogy completion accuracy | Mixed |
| Add multiplicative noise to co-occurrence entries (ablation) | PMI embeddings · Synthetic analogy tasks | analogy accuracy | Hurt |
| Retain only a fraction f=0.15 of vocabulary words, with multiplicative noise (ablation) | PMI embeddings · Synthetic analogy tasks, d=12 | analogy accuracy | No clear effect |
| Prune co-occurrences of word pairs matching an analogy relation (ablation) | Wikipedia PMI embeddings · Wikipedia analogy tasks | analogy accuracy | Hurt |
| Prune co-occurrences along a semantic dimension (ablation) | Synthetic PMI embeddings · Analogy tasks, d=8 | analogy accuracy | No clear effect |
| Use broadly distributed rather than narrowly distributed semantic strengths s_k with raw M (ablation) | Raw-M embeddings · Synthetic analogy tasks | analogy accuracy | Hurt |
| Increase K beyond the semantic dimensions in the presence of noise (ablation) | Noisy PMI embeddings · Synthetic analogy tasks | analogy accuracy | Hurt |
| Allow correlations between semantic attributes (ablation) | PMI model · Linear analogy structure | analogy accuracy | No clear effect |
Surprising findings
- Analogies remain exact in the PMI model even when all co-occurrence pairs for one attribute (e.g., king-queen, man-woman) are removed, because the perturbation is small relative to semantic eigenvalues.
- Perfect analogy accuracy for the M target is fragile and requires a narrow distribution of s_k.
Limitations
- The model is a great simplification: polysemy and hierarchical attribute structure are not captured.
- Assumes all 2^d attribute combinations exist as words and attributes act independently (relaxed only partly in appendix A5).
Experimental setup
Datasets: Wikipedia co-occurrence matrix Mikolov analogy benchmark (19,544 analogies, 13 families)
Benchmarks: Mikolov analogy benchmark
Metrics: top-1 analogy accuracy
Baselines: Word2Vec GloVe Arora et al. random walk model
Builds on: Word2Vec GloVe Levy and Goldberg matrix factorization view of word embeddings
Closest papers by meaning
- Analogies Explained: Towards Understanding Word Embeddings (ICML 2019 · similarity 0.91)
- Joint Word Representation Learning Using a Corpus and a Semantic Lexicon (AAAI 2016 · similarity 0.88)
- By Tying Embeddings You Are Assuming the Distributional Hypothesis (ICML 2024 · similarity 0.87)
- Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models (NeurIPS 2025 · similarity 0.86)
- On the Origins of Linear Representations in Large Language Models (ICML 2024 · similarity 0.86)
- The Geometry of Categorical and Hierarchical Concepts in Large Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.