Research map / Architectures and efficiency / RNNs and normalization
Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace independent vocabulary embeddings with sparse anchor combinations | CNN text classification · AG-News test | accuracy: 91.6 -> 91.0 | No clear effect |
| use 10 frequency-selected anchors instead of 50 (ablation) | ANT · AG-News test | accuracy: 89.5 -> 91.0 | Helped |
| initialize 10 anchors randomly instead of by frequency (ablation) | ANT · AG-News test | accuracy: 91.0 -> 90.5 | Hurt |
| train sparse embeddings end-to-end rather than compress after training | language modeling · WikiText-103 test | perplexity: 73.7 -> 38.4 | Helped |
| use 500 rather than 1,000 random anchors (ablation) | ANT · WikiText-103 test | perplexity: 38.4 -> 54.2 | Hurt |
| use frequency rather than random anchor initialization (ablation) | ANT · PTB test | perplexity: 96.5 -> 70.0 | Helped |
| use 1,000 rather than 2,000 random anchors (ablation) | ANT · PTB test | perplexity: 71.5 -> 73.1 | Hurt |
| replace full matrix-factorization embeddings with sparse anchor combinations | MovieLens 25M recommendation · MovieLens 25M | MSE: 0.665 -> 0.617 | Helped |
| reduce ANT anchors from 10/15 to 5/5 for users/items (ablation) | ANT recommendation · MovieLens 25M | MSE: 0.617 -> 0.651 | Hurt |
| automatically select 8 anchors rather than use ANT's 10/15 setting | MovieLens 25M recommendation · MovieLens 25M | MSE: 0.617 -> 0.637 | Hurt |
| replace full matrix-factorization embeddings with sparse anchor combinations | Amazon Electronics recommendation · Amazon Electronics | MSE: 1.524 -> 1.422 | Helped |
| reduce ANT anchors from 20/8 to 8/3 for users/items (ablation) | ANT recommendation · Amazon Electronics | MSE: 1.422 -> 1.529 | Hurt |
Closest papers by meaning
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.87)
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.87)
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.87)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.86)
- Layer by Layer: Uncovering Hidden Representations in Language Models (ICML 2025 · similarity 0.86)
- Learning and Transferring Sparse Contextual Bigrams with Linear Transformers (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.