Research map / Architectures and efficiency / RNNs and normalization
Further results on latent discourse models and word embeddings
Sammy Khalife, Douglas Gonçalves, Youssef Allouah, Leo Liberti
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| check linear PMI vs scalar product relation (Eq. 25) with SN vectors | Arora et al. latent discourse model · corpus 1, d=50, window 2 | Pearson correlation: 0.04 (R2 0.002) | Hurt |
| increase dimension from 50 to 300 for Eq. 25 on corpus 1 | SN embeddings, window 2 · corpus 1 | Pearson correlation: 0.04 -> 0.24 | Mixed |
| increase dimension from 50 to 300 for Eq. 23 | SN embeddings, window 2 · corpus 1 | Pearson correlation: 0.73 -> 0.76 | Improved |
| increase dimension from 50 to 300 for Eq. 24 (full regression) | SN embeddings, window 2 · corpus 1 | Pearson correlation: 0.84 -> 0.91 | Improved |
| restrict regression to the 50 most frequent words (partial) for Eq. 24 (ablation) | SN embeddings, window 10 · corpus 1, d=50 | slope: 0.120 -> 0.049 | Improved |
| increase window size from 2 to 10 for Eq. 25 | SN embeddings, d=50 · corpus 1 | Pearson correlation: 0.04 -> 0.09 | Mixed |
| use GloVe instead of SN vectors for Eq. 25 | word embeddings, window 2, d=50 · corpus 2 | Pearson correlation: 0.21 -> 0.05 | Hurt |
| use larger corpus (corpus 3 vs corpus 1) for Eq. 25 with SN | SN embeddings, d=50, window 2 · Wikipedia extracts | Pearson correlation: 0.04 -> 0.21 | Improved |
| restrict word-rank range to top 5006 for the PMI-scalar-product relation (ablation) | SN embeddings · corpus 2, d=50, window 2 | fit quality of Eq. 25 | Improved |
| change concentration assumption from Gaussian to symmetric bounded vectors | partition function concentration · theory | concentration of Zc | No clear effect |
Closest papers by meaning
- By Tying Embeddings You Are Assuming the Distributional Hypothesis (ICML 2024 · similarity 0.88)
- On the Emergence of Linear Analogies in Word Embeddings (NeurIPS 2025 · similarity 0.87)
- Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models (NeurIPS 2025 · similarity 0.87)
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.86)
- Token Embeddings Violate the Manifold Hypothesis (NeurIPS 2025 · similarity 0.86)
- L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
