Research map / Architectures and efficiency / Pruning and efficient tuning
Liger: Linearizing Large Language Models to Gated Recurrent Structures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Liger linearizes pretrained Transformer LLMs (Llama-3 1B-8B, Mistral-7B) into gated linear recurrent models by repurposing the key projection to build gates, applying softmax as feature map, and LoRA fine-tuning on 50K Alpaca samples (~0.02B tokens), plus an intra-layer hybrid "Liger Attention" with sliding window softmax.
Problem
Pretraining linear recurrent LLMs from scratch is costly, and existing linearization methods add extra feature-map and gating modules that need training from scratch and fine-tuning, diverging from the pretrained softmax attention.
Approach
Construct gates by pooling the pretrained key projection (parameter-free), use softmax as the feature map so no new parameters are added, and fine-tune end-to-end with LoRA; combine a gated linear recurrence with narrow sliding-window softmax attention (Liger Attention) at equal weights.
Key results
- Liger-GLA-Llama-3-8B reaches 67.6 average (72.4 without MMLU) vs 71.7 for Llama-3-8B using 0.02B tokens, versus LoLCATs at 0.04B tokens scoring 65.4.
- Liger-GLA-Llama-3.2-1B averages 52.9 vs 55.1 for Llama-3.2-1B and 51.1 for LoLCATs.
- Ablation: removing SWA drops Llama-3-8B linearized average from 67.6 to 54.2; replacing feature map with learnable module drops it to 43.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Liger-GLA linearization instead of LoLCATs | Llama-3-8B · Llama-3-8B benchmarks | Avg.: 65.5 -> 67.6 | Helped |
| Replace GLA with HGRN2 recurrent structure (ablation) | Liger · Llama-3-8B benchmarks | Avg.: 67.6 -> 64.0 | Hurt |
| Replace GLA with GSA recurrent structure (ablation) | Liger · Llama-3-8B benchmarks | Avg.: 67.6 -> 65.6 | Hurt |
| Use pooled gate construction instead of a randomly initialized gate projection (ablation) | Liger-GLA · Llama-3-8B benchmarks | Avg.: 63.8 -> 67.6 | Helped |
| Add a feature map module (ablation) | Liger-GLA · Llama-3-8B benchmarks | Avg.: 40.2 -> 67.6 | Helped |
| Add gating to pure linear attention (ablation) | Liger-GLA · Llama-3-8B benchmarks | Avg.: 66.1 -> 67.6 | Helped |
| Add LoRA fine-tuning (ablation) | Liger-GLA · Llama-3-8B benchmarks | Avg.: 61.7 -> 67.6 | Helped |
| Add sliding-window attention (ablation) | Liger-GLA · Llama-3-8B benchmarks | Avg.: 54.2 -> 67.6 | Helped |
| Add gated linear attention (ablation) | Liger-GLA · Llama-3-8B benchmarks | Avg.: 66.2 -> 67.6 | Helped |
Surprising findings
- Removing the GLA gating (w/o GLA) leaves the average nearly unchanged (66.2 vs 67.6 is a small drop; no-MMLU 72.0 vs 72.4), suggesting the gate contributes little in this setup.
- Gate construction from the query or value projection underperforms key-based pooling, though value pooling gives the highest MMLU score (43.7).
Limitations
- Evaluation is limited to short-context multiple-choice benchmarks (PiQA, ARC, HellaSwag, WinoGrande, MMLU) with no long-context or generation evaluation.
- Remaining gap to base model (about 4.1 points average on 8B, larger at 1B) and reliance on the 0.02B-token Alpaca fine-tuning.
- Gating is parameter-free pooling of the key projection, and the paper gives limited theoretical justification for it.
Experimental setup
Datasets: Cleaned Alpaca
Benchmarks: PiQA ARC-e ARC-c HellaSwag WinoGrande MMLU
Metrics: accuracy average accuracy validation perplexity decoding latency GPU memory
Baselines: SUPRA MambaInLlama LoLCATs GLA Mamba RWKV-6 Zamba Zamba2
Builds on: Gated Linear Attention LoLCATs LoRA Mamba2 in Llama
Closest papers by meaning
- LoLCATs: On Low-Rank Linearizing of Large Language Models (ICLR 2025 · similarity 0.90)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.89)
- Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.88)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.88)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.88)
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.