Research map / Architectures and efficiency / Pruning and efficient tuning
BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BLAST, a block-level adaptive structured matrix with shared row/column bases and per-block diagonal couplings, used for training from scratch or compressing pretrained ViT, GPT-2, DiT and Llama-7B weights.
Problem
Dense matrix-vector operations in large DNNs (e.g., Llama-7B, DiT-XL, ViT, GPT-2) are costly at inference, and existing structured matrices (low-rank, Monarch, block-diagonal, GBLR) either misalign with the true weight structure or do not accelerate well on GPUs.
Approach
A b×b block matrix where each block is U_i diag(s_ij) V_j^T, with left/right bases shared across block rows/columns and only a small diagonal coupling vector per block; this subsumes low-rank, block-diagonal and BLR structures, is trainable by gradient descent, and admits a preconditioned gradient factorization for compressing pretrained weights.
Key results
- On Llama-7B at 50% compression with re-training on 0.49B tokens, BLAST16 reaches 14.21 WikiText-2 perplexity and 56.22% average zero-shot accuracy versus 26.33 and 48.40 for low-rank (original: 9.37 and 66.07).
- BLAST16 at 20% compression on Llama-7B without re-training keeps 12.13 perplexity and 62.94% accuracy, while low-rank and Monarch lose far more (23.67 and 47.18 perplexity).
- The abstract reports 70% complexity reduction for ViT and 40% for GPT-2 when training with BLAST weights from scratch, and a 12-15% (20% CR) to 32-35% (50% CR) Llama-7B runtime reduction on an A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BLAST16 instead of Low-Rank for 20% compression without re-training | Llama-7B · WikiText-2 | perplexity: 23.67 -> 12.13 | Helped |
| Use BLAST16 instead of Low-Rank for 20% compression without re-training | Llama-7B · Average zero-shot accuracy | accuracy: 59.57 -> 62.94 | Helped |
| Use BLAST16 instead of Low-Rank for 50% compression with re-training | Llama-7B · WikiText-2 | perplexity: 26.33 -> 14.21 | Helped |
| Use BLAST16 instead of Low-Rank for 50% compression with re-training | Llama-7B · Average zero-shot accuracy | accuracy: 48.40 -> 56.22 | Helped |
| Use BLAST16 instead of Monarch or Block-Diagonal at 50% compression with re-training | Llama-7B · WikiText-2 | perplexity: 7.53e5 -> 14.21; 5.21e6 -> 14.21 | Helped |
| Use BLAST16 instead of Monarch or Block-Diagonal at 50% compression with re-training | Llama-7B · Average zero-shot accuracy | accuracy: 35.03 -> 56.22; 34.86 -> 56.22 | Helped |
| Add re-training after BLAST16 compression (ablation) | Llama-7B at 50% compression · Common-sense reasoning benchmarks | Average zero-shot accuracy | Helped |
| Add preconditioning to BLAST factorization (ablation) | Overparameterized low-rank matrix factorization · Synthetic 256 × 256 target matrix, true rank 8 | Factorization error | Helped |
Surprising findings
- Monarch and Block-Diagonal compressed Llama-7B at 50% with re-training collapse to perplexities in the 10^5 to 10^6 range, showing non-adaptive structures fail at high compression.
- The authors note BLAST's runtime gains come mainly from parameter reduction (memory bandwidth bound) rather than FLOP reduction.
Limitations
- The per-layer rank budget is uniform rather than adaptive, which the authors identify as a limitation that could be improved.
- Not evaluated on tiny (<100M parameter) or very large (>10B parameter) DNNs.
- The three-step matrix-vector product may reduce hardware parallelism; runtime gains rely on no custom kernels.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k WikiText-103 SlimPajama WikiText-2 PIQA HellaSwag WinoGrande BoolQ OpenBookQA ARC-easy
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k WikiText-103 WikiText-2 PIQA HellaSwag WinoGrande BoolQ OpenBookQA ARC-easy ARC-challenge
Metrics: top-1 accuracy perplexity FLOPs FID sFID Inception Score zero-shot accuracy runtime (seconds)
Baselines: Low-Rank (SVD) Monarch Block-Diagonal Gaudi-GBLR Pixelfly SVD low-rank compression
Builds on: Monarch Gaudi-GBLR Pixelfly DiT ViT
Closest papers by meaning
- Compute Better Spent: Replacing Dense Layers with Structured Matrices (ICML 2024 · similarity 0.89)
- Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices (NeurIPS 2024 · similarity 0.86)
- Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networks (ICLR 2024 · similarity 0.85)
- Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples (NeurIPS 2025 · similarity 0.84)
- An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networks (ICML 2025 · similarity 0.84)
- MoDeGPT: Modular Decomposition for Large Language Model Compression (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.