Research map / Architectures and efficiency / Attention and transformers
Token Distillation: Attention-Aware Input Embeddings for New Tokens
Konstantin Dobler, Desmond Elliott, Gerard de Melo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hidden-state distillation instead of subtoken mean init | subtoken mean initialization · biomedical benchmarks avg | accuracy: 60.8 -> 64.6 | Improved |
| use token distillation instead of next-token prediction tuning | NTP · biomedical benchmarks avg | accuracy: 63.0 -> 64.6 | Improved |
| use token distillation instead of ZeTT hyper-network | ZeTT · biomedical Llama3-8B | accuracy: 66.1 -> 67.3 | Improved |
| freeze original embeddings and optimize only new tokens (NTP) (ablation) | NTP tune all embeddings · biomedical benchmarks avg | accuracy: 59.8 -> 63.0 | Improved |
| sum NTP loss with distillation loss (ablation) | Token Distillation · biomedical benchmarks avg | accuracy: 64.6 -> 64.3 | Hurt |
| scale NTP loss by autoscaling factor alpha (ablation) | Token Distillation + NTP · biomedical benchmarks avg | accuracy: 64.3 -> 64.7 | Improved |
| add autoscaled NTP to token distillation (ablation) | Token Distillation · biomedical Llama3.2-3B | accuracy: 56.2 -> 57.9 | Improved |
| generate contexts with model instead of retrieving from corpus (ablation) | Token Distillation · biomedical benchmarks avg | accuracy: 64.6 -> 64.3 | Hurt |
| start from subtoken mean instead of random init (ablation) | Token Distillation · biomedical Mistral-7B | accuracy: 62.5 -> 63.5 | Improved |
| start from ZeTT embeddings instead of subtoken mean (ablation) | Token Distillation · biomedical Llama3-8B | accuracy: 67.3 -> 67.9 | Improved |
| use KL or logit-MSE instead of hidden-state MSE (ablation) | Token Distillation · biomedical benchmarks | accuracy | No clear effect |
| target an earlier layer than the last (ablation) | Token Distillation · biomedical Llama3-8B-Instruct | accuracy | Improved |
Closest papers by meaning
- Attention with Trained Embeddings Provably Selects Important Tokens (NeurIPS 2025 · similarity 0.88)
- Model-Aware Tokenizer Transfer (NeurIPS 2026 · similarity 0.87)
- Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching (NeurIPS 2025 · similarity 0.87)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.86)
- Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings (NeurIPS 2025 · similarity 0.86)
- Leviathan: Decoupling Input and Output Representations in Language Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
