Research map / Architectures and efficiency / RNNs and normalization
By Tying Embeddings You Are Assuming the Distributional Hypothesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes the distributional hypothesis via semantic and conditional equivalence of symbols, proves input/output embedding properties under optimality, and uses a synthetic EXor task to argue weight tying helps only when the distributional hypothesis holds.
Problem
Why do input and output embeddings of language models organize into semantic spaces, and why is input-output weight tying effective? The paper lacks a formal account of this.
Approach
Defines semantic equivalence and conditional equivalence of symbols, formalizes the distributional hypothesis as their equivalence, and proves that optimal models must give similar input embeddings to semantically similar symbols and similar output embeddings to conditionally similar symbols. From this, tying is beneficial only when the distributional hypothesis holds.
Key results
- On the EXor task, 1A and 1B embeddings converge faster in the tied model than in the untied model, consistent with the theory when the distributional hypothesis holds.
- The tied model is slightly lower in test accuracy than the untied model on EXor, attributed to instability where the distributional hypothesis fails for 0A/0B.
- Untied output embeddings show only the conditionally equivalent pair 1A-1B approaching distance 0, as predicted by Theorem 4.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| tie input and output embeddings | EXor model · EXor test set | accuracy | Hurt |
| tie input and output embeddings | EXor model · EXor, 1A–1B embedding pair | Euclidean embedding distance | Helped |
| tie input and output embeddings | EXor model · EXor, 0A–0B embedding pair | Euclidean embedding distance | Mixed |
Surprising findings
- Tying embeddings slightly hurt test accuracy on EXor relative to untied embeddings, even though tying often helps in practice.
Limitations
- Theorems rely on strong assumptions: a single self-attention layer, optimal model, and a basis condition on embeddings.
- Experiments are limited to a small synthetic EXor task with a tiny vocabulary and only one small architecture.
- Degree to which the distributional hypothesis holds in real data is hard to measure, limiting practical applicability.
Experimental setup
Datasets: EXor (synthetic extended XOR masked-LM dataset)
Benchmarks: EXor
Metrics: test accuracy Euclidean distance between embeddings
Baselines: untied input-output embeddings
Builds on: Harris (1954) distributional hypothesis Press & Wolf (2017) weight tying Inan et al. (2016) tied embeddings Tian et al. (2023)
Closest papers by meaning
- Uncovering Meanings of Embeddings via Partial Orthogonality (NeurIPS 2023 · similarity 0.88)
- A Generative Model of Words and Relationships from Multiple Sources (AAAI 2016 · similarity 0.88)
- Understanding the Semantic Content of Sparse Word Embeddings Using a Commonsense Knowledge Base (AAAI 2020 · similarity 0.88)
- Token Embeddings Violate the Manifold Hypothesis (NeurIPS 2025 · similarity 0.88)
- Semantic Structure-Based Word Embedding by Incorporating Concept Convergence and Word Divergence (AAAI 2018 · similarity 0.87)
- On the Emergence of Linear Analogies in Word Embeddings (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.