Research map / Architectures and efficiency / Attention and transformers
STEM: SCALING TRANSFORMERS WITH EMBEDDING MODULES
Ranajoy Sadhukhan, Sheng Cao, Harry Dong, Changsheng Zhao, Attiano Purpura-Pontoniere, Yuandong Tian, Zechun Liu, Beidi Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace 1/3 of FFN up-projections with STEM embeddings | MobileLLM-350M · Zero-shot commonsense avg, 350M | avg accuracy: 49.72 -> 50.90 | Improved |
| STEM-1/3 vs Hash-MoE at same total params | MobileLLM-350M · Zero-shot commonsense avg, 350M | avg accuracy: 50.58 -> 50.90 | Improved |
| replace 1/2 instead of 1/3 of FFN layers (ablation) | STEM MobileLLM-350M · Zero-shot commonsense avg, 350M | avg accuracy: 50.90 -> 54.20 | Improved |
| replace all FFN layers instead of 1/2 (ablation) | STEM MobileLLM-350M · Zero-shot commonsense avg, 350M | avg accuracy: 54.20 -> 53.43 | Mixed |
| replace up-projection instead of gate-projection (ablation) | STEM MobileLLM-350M · Zero-shot commonsense avg, 350M | avg accuracy: 49.10 -> 50.90 | Improved |
| keep up-projection with additive STEM embedding (STEM-dagger) (ablation) | STEM MobileLLM-350M · Zero-shot commonsense avg, 350M | avg accuracy: 50.90 -> 50.60 | No clear effect |
| apply STEM-1/3 at 1B scale | Llama3.2-1B · Zero-shot commonsense avg | avg accuracy: 55.82 -> 56.63 | Improved |
| midtrain 1B STEM | Llama3.2-1B · GSM8K | accuracy: 44.2 -> 46.4 | Improved |
| midtrain 1B STEM | Llama3.2-1B · MMLU | accuracy: 29.92 -> 32.38 | Improved |
| apply STEM on ARC-Challenge with 1/2 replacement (ablation) | MobileLLM-350M · ARC-Challenge | accuracy: 30.55 -> 40.00 | Improved |
| apply STEM-1/3 on SIQA | MobileLLM-350M · SIQA | accuracy: 41.10 -> 39.76 | Hurt |
| apply STEM at 1B | Llama3.2-1B · BoolQ | accuracy: 64.21 -> 61.66 | Hurt |
Closest papers by meaning
- TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (ICLR 2025 · similarity 0.88)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.85)
- Scaling Embedding Layers in Language Models (NeurIPS 2025 · similarity 0.85)
- Dense Structural Compression of Transformers via Gauge-Correct Channel Removal (NeurIPS 2026 · similarity 0.85)
- StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training (NeurIPS 2025 · similarity 0.84)
- Distilling Sequential Computation in Transformer Language Models (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
