Research map / Large language models / Language, safety and interpretability
Learn To be Efficient: Build Structured Sparsity in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Learn-To-be-Efficient (LTE), a two-stage training method that MoEfies FFN layers with sigmoid threshold routing plus an efficiency loss and separability loss, yielding more structured activation sparsity in both ReLU and non-ReLU LLMs.
Problem
LLM inference is costly in FFN layers; existing MoEfication methods exploit only naturally occurring activation sparsity, which is low in soft-activation models like LLaMA, and top-k softmax routing hurts accuracy when converting FFNs to MoE.
Approach
Group FFN neurons into experts via balanced K-means on weights, route with independent sigmoid scores and a fixed threshold, and train the router and model jointly with an efficiency loss (penalizing router output magnitude) and a threshold-separability loss, followed by a discrete-routing adaptation stage.
Key results
- LTE gives 1.83x-2.59x FLOPs speed-up on LLaMA-7B across XSum, E2E and WikiText-103 within the stated quality tolerance (e.g., 4.65 GFLOPs/token on E2E vs 12.06 dense).
- With a custom Triton kernel, LTE reduces LLaMA2-7B end-to-end generation latency by about 25% at around 50% FFN sparsity on a single 3090Ti.
- On WikiText103 with overall 52% sparsity, LTE reaches perplexity 5.95 versus 6.87 for the best Wanda variant at 50% sparsity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LTE instead of baseline sparse methods | LLaMA-7B · XSum | GFLOPs per token: 12.06 -> 5.38 | Helped |
| Use LTE instead of baseline sparse methods | LLaMA-7B · E2E | GFLOPs per token: 12.06 -> 4.65 | Helped |
| Use LTE instead of baseline sparse methods | LLaMA-7B · WikiText-103 | GFLOPs per token: 12.06 -> 6.59 | Helped |
| Use LTE instead of Wanda at similar overall sparsity | LLaMA-7B · WikiText103 | perplexity: 6.87 -> 5.95 | Helped |
| Replace parameter K-means grouping with random neuron grouping (ablation) | LTE · WikiText-103 (GPT2-Medium) | performance | Hurt |
| Replace parameter K-means grouping with co-activation graph split (ablation) | LTE · WikiText-103 (GPT2-Medium) | performance | No clear effect |
| Increase efficiency-loss coefficient η (ablation) | LTE · Evaluated models and tasks | sparsity | Helped |
| Add separability loss (λ from 0 to 0.5) (ablation) | LTE · WikiText-103 (GPT2-Medium) | perplexity | Helped |
| Increase separability-loss coefficient λ beyond 0.5 (ablation) | LTE · WikiText-103 (GPT2-Medium) | perplexity | No clear effect |
| Replace LTE-trained routers with random routers (ablation) | LTE · WikiText-103 (GPT2-Medium) | performance at different sparsity levels | Hurt |
| Replace GeLU with ReLU (ablation) | RoBERTa · MRPC | accuracy: around 10% accuracy drop | Hurt |
| Use threshold-based adaptive LTE expert selection rather than fixed expert counts | LLaMA2-7B with custom Triton kernel · End-to-end generation | latency: 25\% | Helped |
Surprising findings
- Replacing GeLU with ReLU in RoBERTa causes around 10% accuracy drop on MRPC, so the ReLU-substitution assumption does not hold universally.
- Noisy top-k softmax routing shows severe accuracy drops even at very low sparsity because expert scores collapse to one dominant expert per layer.
Limitations
- Requires additional fine-tuning to obtain contextual sparse models, unlike training-free methods.
- Weaker sparsity-performance trade-off on instruction tuning/MMLU, possibly because only small instruction data was used and not pretraining data.
- Compute limits prevented training LTE on pretraining or RLHF.
Experimental setup
Datasets: GLUE WikiText-103 XSum E2E Tulu
Benchmarks: GLUE E2E XSum WikiText-103 MMLU
Metrics: accuracy Matthews correlation Pearson correlation ROUGE-L perplexity GFLOPs per token wall-clock latency
Baselines: MoEfication Deja Vu R-LLaMA+MoE Wanda σ-MoE
Builds on: MoEfication Deja Vu noisy top-k softmax gating Wanda
Closest papers by meaning
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.93)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.92)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.92)
- The Sparsity Whisperer (NeurIPS 2026 · similarity 0.91)
- ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models (ICLR 2024 · similarity 0.91)
- Training-Free Activation Sparsity in Large Language Models (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.