Research map / Architectures and efficiency / Quantization
Equilibrium Language Models
Yikun Jiang, Huanyu Wang, Tianhong Ding, Wenhu Zhang, Yiming Wu, Hanbin Zhao, John C.S. Lui
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add One-Step KV cache (vs Multi-Step) (ablation) | ELM Qwen2.5-1.5B-Instruct M=8 · GSM8K | accuracy: 71.4 -> 71.8 | No clear effect |
| add One-Step KV cache (vs Multi-Step) (ablation) | ELM Qwen2.5-1.5B-Instruct M=14 · KV cache memory (4K context) | KV (MB): 114.8 -> 61.6 | Improved |
| add One-Step KV cache (vs Multi-Step) (ablation) | ELM Qwen2.5-1.5B-Instruct M=8 · KV cache memory (4K context) | KV (MB): 114.8 -> 86.0 | Improved |
| use GPPO to pick start layer instead of cosine similarity (ablation) | ELM Qwen2.5-1.5B-Instruct · MBPP | pass@10: 65.4 -> 70.6 | Improved |
| use GPPO to pick start layer instead of perplexity (ablation) | ELM Qwen2.5-1.5B-Instruct · MBPP | pass@10: 66.6 -> 70.6 | Improved |
| use transformer layer instead of FFN as fixed-point module (ablation) | ELM Qwen2.5-1.5B-Instruct · MMLU | accuracy: 51.6 -> 54.0 | Improved |
| initialize Wh as identity instead of random (ablation) | ELM Qwen2.5-1.5B-Instruct · MBPP | pass@10: 69.8 -> 70.6 | Improved |
| train with SJFB (ablation) | ELM Qwen2.5-1.5B-Instruct · training cost (math) | GPU-days: 7.1 -> 6.1 | Improved |
| prune one continuous group instead of two disjoint groups (M=14) (ablation) | ELM Qwen2.5-1.5B-Instruct · MBPP | pass@10: 58.4 -> 65.4 | Improved |
| prune one continuous group instead of two disjoint groups (M=14) (ablation) | ELM Qwen2.5-1.5B-Instruct · HumanEval | pass@10: 56.7 -> 58.5 | Improved |
| use Anderson acceleration instead of simple iteration (M=14, delta 0.5) | ELM Qwen2.5-1.5B-Instruct · GSM8K iterations | iterations: 12.25 -> 7.52 | Improved |
| use Broyden solver instead of simple iteration (M=8) | ELM Qwen2.5-1.5B-Instruct · GSM8K | accuracy: 71.8 -> 69.8 | Hurt |
Closest papers by meaning
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.90)
- Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.90)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.89)
- Let LLM Tell What to Prune and How Much to Prune (ICML 2025 · similarity 0.89)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.89)
- You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
