Research map / Large language models / Reasoning and chain of thought
LoLCATs: On Low-Rank Linearizing of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoLCATs, a two-step method that linearizes LLMs by first training linear/sliding-window attention to match softmax outputs (attention transfer) and then applying LoRA, scaling to Llama 3.1 70B and 405B with tiny token budgets.
Problem
Linearizing pretrained Transformer LLMs into subquadratic attention usually degrades quality and needs billions of tokens and full-parameter training, and has not scaled beyond ~8B parameters.
Approach
Train learnable linear attention feature maps (plus a sliding-window softmax term) to match each layer's softmax attention outputs via MSE (attention transfer), then recover quality with LoRA on attention projections; apply it block-by-block to make 70B/405B feasible.
Key results
- Llama 3 8B LoLCATs reaches 70.7 average on LM Eval with MMLU (vs 72.0 Transformer) and 52.8 MMLU, 20+ pts above prior linearized models using only 0.04B tokens.
- Linearized Llama 3.1 70B reaches 67.7 5-shot MMLU vs 78.8 original and 28.7 without attention transfer.
- Linearized Llama 3.1 405B reaches 72.2 5-shot MMLU vs 82.98 original and 33.86 without attention transfer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add attention transfer before LoRA adjusting (ablation) | Llama 3 8B and Mistral 7B linearized with T2R or Hedgehog · Alpaca validation | perplexity: 1539.39 -> 79.33; 2448.01 -> 60.86; 2497.13 -> 32.78; 561.47 -> 18.94 | Helped |
| Add sliding-window attention and attention transfer (ablation) | Hedgehog linearizing of Llama 3 8B · LM Eval | average: 44.20 -> 70.66 | Helped |
| Add attention transfer without sliding-window attention (ablation) | Hedgehog linearizing of Llama 3 8B · LM Eval | average: 44.20 -> 55.32 | Helped |
| Add sliding-window attention and attention transfer (ablation) | T2R linearizing of Llama 3 8B · LM Eval | average: 38.84 -> 68.28 | Helped |
| Add sliding-window attention and attention transfer (ablation) | T2R linearizing of Llama 3 8B · MMLU (5-shot) | accuracy: 23.20 -> 40.70 | Helped |
| Retain the first 16 softmax-attention layers and use attention transfer only (ablation) | Llama 3 8B LoLCATs · MMLU (5-shot) | accuracy: 52.8 -> 65.8 | Helped |
| Use LoLCATs rather than prior linearized Mistral 7B SUPRA | Mistral 7B · MMLU (5-shot) | accuracy: 34.2 -> 51.4 | Helped |
| Use LoLCATs rather than linearizing without attention transfer | Llama 3.1 70B · MMLU (5-shot) | accuracy: 28.74 -> 67.70 | Helped |
| Use LoLCATs rather than linearizing without attention transfer | Llama 3.1 405B · MMLU (5-shot) | accuracy: 33.86 -> 72.20 | Helped |
Surprising findings
- Attention transfer plus LoRA on only attention projections nearly matches full-parameter linearizing baselines while training under 0.2% of parameters.
- Jointly training attention transfer across all 126 layers of Llama 3.1 405B fails to viably linearize it, motivating block-wise training.
Limitations
- Evaluated only on short-context tasks (1024-token sequences) and zero-shot LM Eval plus 5-shot MMLU; long-context behavior left to future work.
- Uses only vanilla linear + sliding window attention without decay or gating terms from newer architectures.
- Remaining MMLU gap to Transformers (e.g., 52.8 vs 66.6 for Llama 3 8B) persists in the default setting.
Experimental setup
Datasets: Alpaca (cleaned) RedPajama-Data-1T-Sample
Benchmarks: LM Eval Harness (PiQA, ARC-e, ARC-c, HellaSwag, Winogrande) MMLU (5-shot)
Metrics: accuracy accuracy normalized average score validation perplexity throughput memory
Baselines: Hedgehog T2R SUPRA Mamba2-Llama 3 RWKV-6 Mamba Griffin Zamba StripedHyena FlashAttention-2 MOHAWK
Builds on: Hedgehog T2R LoRA FlashAttention-2 MOHAWK
Closest papers by meaning
- Liger: Linearizing Large Language Models to Gated Recurrent Structures (ICML 2025 · similarity 0.90)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.90)
- Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs (CVPR 2024 · similarity 0.90)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.90)
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.90)
- Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.