Research map / Large language models / Language, safety and interpretability
Sparser, Faster, Lighter Transformer Language Models
Edoardo Cetin, Stefano Peluchetti, Emilio Castillo, Akira Naruse, Mana Murakami, Llion Jones
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add L1 regularization (2e-5) and sparse TwELL kernels | Transformer++ 0.5B · FineWeb, 0.5B | forward execution (input tokens/ms): 410 -> 480 | Improved |
| add L1 regularization and sparse kernels | Transformer++ 1B · FineWeb, 1B | forward execution (input tokens/ms): 185 -> 219 | Improved |
| add L1 regularization and sparse kernels | Transformer++ 1.5B · FineWeb, 1.5B | forward execution (input tokens/ms): 119 -> 141 | Improved |
| add L1 regularization and sparse kernels | Transformer++ 2B · FineWeb, 2B | forward execution (input tokens/ms): 87.8 -> 106 | Improved |
| add L1 regularization and sparse kernels | Transformer++ 2B · FineWeb, 2B | energy per token (mJ): 7.85 -> 6.51 | Improved |
| add L1 regularization and sparse kernels | Transformer++ 0.5B · FineWeb, 0.5B | training step (input tokens/ms): 97.3 -> 95.9 | No clear effect |
| add L1 regularization and sparse kernels | Transformer++ 2B · FineWeb, 2B | training step (input tokens/ms): 22.4 -> 27.3 | Improved |
| add L1 regularization and sparse kernels | Transformer++ 1.5B · FineWeb, 1.5B | peak memory (GB): 62.8 -> 45.1 | Improved |
| add L1 regularization and sparse kernels | Transformer++ 1B · FineWeb, 1B | mean task accuracy (%): 44.6% -> 44.7% | No clear effect |
| add L1 regularization and sparse kernels | Transformer++ 2B · FineWeb, 2B | mean task accuracy (%): 49.1% -> 48.8% | No clear effect |
| raise L1 coefficient up to 3e-5 (ablation) | Transformer++ 1.5B · FineWeb, 1.5B | downstream task performance | No clear effect |
| raise L1 so fewer than 0.5% of neurons are active (ablation) | Transformer++ 1.5B · FineWeb, 1.5B | downstream task performance | Hurt |
Closest papers by meaning
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.92)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.91)
- DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs (NeurIPS 2025 · similarity 0.91)
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.91)
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.91)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
