Research map / Large language models / Reasoning and chain of thought
Efficient Large Language Model Inference with Neural Block Linearization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Neural Block Linearization (NBL), which replaces selected LLM self-attention layers with closed-form LMMSE linear estimators, ranking layers by a CCA-based NMSE upper bound, with no fine-tuning.
Problem
Reducing the inference cost of transformer LLMs; existing structured pruning (layer/block dropping) often causes large accuracy drops when removing layers.
Approach
Replace attention layers with LMMSE-derived linear maps computed from calibration activations, and select which layers to linearize using a CCA-derived upper bound on the normalized MSE (lowest bound = most linearizable), requiring no gradient training.
Key results
- On Mistral-7B, Attn NBL-8 keeps average accuracy at 70.0 (baseline 70.2) at 2.9% sparsity, versus 69.4 for Attn DROP-8.
- On DeepSeek-R1-Distill-Llama-8B, applying NBL to 12 attention layers gives 32% throughput gain (1.32x) with average accuracy 62.4 vs 63.6 baseline.
- On Llama-3.1-8B, Block NBL-12 reaches 50.4 average accuracy at 34.6% sparsity, above Block DROP-12 (43.9) and SLEB-12 (42.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace 8 attention layers with linear approximations rather than dropping them | Mistral-7B · Reasoning benchmarks | Avg. accuracy: 69.4 -> 70.0 | Helped |
| replace 8 attention layers with linear approximations | Mistral-7B · Reasoning benchmarks | Avg. accuracy: 70.2 -> 70.0 | Hurt |
| replace 8 attention layers with linear approximations | Llama-3.1-8B · Reasoning benchmarks | Avg. accuracy: 70.2 -> 69.8 | Hurt |
| replace 8 attention layers with linear approximations instead of dropping them | DeepSeek-R1-Distill-Llama-8B · Reasoning benchmarks | Avg. accuracy: 63.5 -> 63.2 | Hurt |
| replace 32 attention layers with linear approximations instead of dropping them | Llama-3.1-70B · Reasoning benchmarks | Avg. accuracy: 75.7 -> 76.0 | Helped |
| select 12 layers with the CCA-bound criterion rather than direct NMSE (ablation) | NBL on Mistral-7B · Reasoning benchmarks | Accuracy: 39.1 -> 68.3 | Helped |
| substitute transformer blocks instead of attention layers (ablation) | NBL · Mistral-7B and Llama-3.1-8B reasoning benchmarks | Avg. accuracy | Hurt |
| apply LoRA fine-tuning to NBL-linearized layers (ablation) | NBL · Reasoning benchmarks | Accuracy | Helped |
| calibrate on C4 versus WikiText-2 (ablation) | NBL · C4 and WikiText-2 validation | Accuracy | No clear effect |
Surprising findings
- Selecting layers by direct NMSE instead of the CCA bound drops Mistral-7B accuracy after replacing 12 layers from 68.3 to 39.1.
- Block-level (whole transformer block) linearization causes much larger accuracy degradation than attention-only linearization.
Limitations
- Linear approximations may fail for highly nonlinear attention layers, so selection is needed; applying to all layers degrades the model.
- Calibration data choice can affect generalization to new domains and tasks.
- Compression is evaluated mainly on reasoning/multiple-choice benchmarks; 70B results use 4-bit AWQ baseline and exclude PruneNet and SliceGPT.
Experimental setup
Datasets: C4 WikiText-2 Pile
Benchmarks: MMLU ARC-e ARC-c BoolQ HellaSwag OBQA PIQA WinoGrande
Metrics: accuracy average accuracy prefill speed throughput sparsity
Baselines: SLEB SliceGPT PruneNet DROP (Block and Attn)
Builds on: SLEB DROP SliceGPT PruneNet AWQ
Closest papers by meaning
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.90)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.89)
- Star Attention: Efficient LLM Inference over Long Sequences (ICML 2025 · similarity 0.89)
- LoLCATs: On Low-Rank Linearizing of Large Language Models (ICLR 2025 · similarity 0.89)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.88)
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.