Research map / Architectures and efficiency / Attention and transformers
DeFINE: Deep Factorized Input Token Embeddings for Neural Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DeFINE to an adaptive-input LSTM, using N=3 | LSTM · WikiText-103 test | perplexity: 44.87 -> 41.17 | Helped |
| Add DeFINE to Transformer-XL with standard input-output layers | Transformer-XL · WikiText-103 test | perplexity: 27.06 -> 24.17 | Mixed |
| Add DeFINE to Transformer-XL using projective mappings | Transformer-XL · WikiText-103 test | perplexity: 28.09 -> 25.72 | Helped |
| Replace AWD-LSTM embeddings with DeFINE | AWD-LSTM · Penn Treebank test | perplexity: 58.8 -> 54.2 | Helped |
| Replace Transformer input-output layers with DeFINE without checkpoint averaging | Transformer · WMT 2014 English-German newstest2017 | BLEU: 25.81 -> 28.25 | Helped |
| Use hierarchical group transformation instead of group linear transformation (ablation) | DeFINE token representations · WikiText-103 test | perplexity: 45.63 -> 40.92 | Helped |
| Add input mixing and direct input connections to HGT (ablation) | HGT · WikiText-103 test | perplexity: 40.92 -> 38.01 | Helped |
| Add the mixer to DeFINE without mixer (ablation) | DeFINE · WikiText-103 test | perplexity: 38.91 -> 38.01 | Helped |
| Increase DeFINE depth from 3 to 7 while increasing dimensions (ablation) | LSTM + DeFINE · WikiText-103 test | perplexity: 41.17 -> 38.01 | Helped |
| Increase DeFINE width k at fixed depth N=7 (ablation) | LSTM + DeFINE · WikiText-103 test | perplexity: 38.01 -> 37.81; 38.01 -> 37.84 | No clear effect |
| Replace DeFINE input skip-connections with residual connections (ablation) | DeFINE · WikiText-103 test | perplexity: 38.01 -> 40.03 | Hurt |
| Remove the Reduce step from Map-Expand-Reduce (ablation) | DeFINE · WikiText-103 test | perplexity: 38.01 -> 38.34 | Hurt |
Closest papers by meaning
- VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models (NeurIPS 2025 · similarity 0.87)
- Model-Aware Tokenizer Transfer (NeurIPS 2026 · similarity 0.87)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.86)
- Leviathan: Decoupling Input and Output Representations in Language Models (NeurIPS 2026 · similarity 0.86)
- Distilling Sequential Computation in Transformer Language Models (NeurIPS 2026 · similarity 0.86)
- Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.