Research map / Architectures and efficiency / RNNs and normalization
Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Generalized Block-Low-Rank (GBLR) matrices whose structure is learned by gradient descent via a Gaussian-Dirichlet frequency-domain mask (Gaudi-GBLR) and proximal gradient descent, improving accuracy-FLOPs trade-offs over low-rank, Monarch, and Pixelfly-style matrices for ViT, MLP-Mixer, and GPT-2.
Problem
Replacing dense weight matrices in DNNs with structured matrices is usually done with hand-designed formats (low-rank, block-sparse, block-low-rank) that are disjoint and non-differentiable in their structure choice, so the best per-layer structure is not learned.
Approach
Define a generalized block-low-rank format as a sum of overlapping rank-1 blocks with learnable widths and locations, parameterize the boxcar masks via a Gaussian-Dirichlet function in the frequency domain so that width and location gradients exist, and learn structure with proximal gradient descent under an l1 cost penalty.
Key results
- On GPT-2 fine-tuned on WikiText103, Gaudi-GBLR reaches perplexity 19.24 at 43.7% relative FLOPs, versus 19.36 for dense and 19.48 for low-rank at 43.75% FLOPs.
- Gaudi-GBLR preserves ViT-Base ImageNet accuracy at about 30% of dense FLOPs, with better accuracy-FLOPs trade-offs than LR and Monarch.
- Learned structures are neither block-sparse nor block-low-rank and allocate unequal budgets across layers (e.g., MLP-FC2 uses 7.96x the FLOPs of Value layers on average).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace dense weight matrices with Gaudi-GBLR matrices | GPT-2 · WikiText103 validation | Perplexity: 19.36 -> 19.24 | Helped |
| replace Low Rank matrices with Gaudi-GBLR matrices | GPT-2 · WikiText103 validation | Perplexity: 19.48 -> 19.24 | Helped |
| replace Monarch matrices with Gaudi-GBLR matrices | GPT-2 · WikiText103 validation | Perplexity: 20.56 -> 19.24 | Helped |
| learn structured matrices with Gaudi-GBLR instead of hand-designed LR or Monarch structures | ViT-Base · ImageNet validation | Accuracy | Helped |
| learn structured matrices with Gaudi-GBLR instead of hand-designed structured matrices | ViT and MLP-Mixer · CIFAR-10/100 | Accuracy | Helped |
Surprising findings
- The learned GBLR structure is neither block-sparse nor block-low-rank, with blocks concentrated in a few regions and ranks varying across layers and matrix types.
Limitations
- Non-integer widths and locations produce spiking errors unless Gaussian smoothing is used, requiring sigma annealing or straight-through estimation.
- Optimization is non-convex and only reaches local optima, relying on an SVD-based initialization.
- Evaluated on relatively small vision and language settings (ViT-Base, GPT-2, CIFAR) rather than large LLMs.
Experimental setup
Datasets: ImageNet CIFAR-10 CIFAR-100 WikiText103
Benchmarks: ImageNet CIFAR-10 CIFAR-100 WikiText103
Metrics: top-1 accuracy perplexity FLOPs
Baselines: Low-Rank Monarch Pixelfly (BSP-LR) Dense ViT-Base ALDS
Builds on: Monarch Pixelfly Dirichlet kernel AdamW
Closest papers by meaning
- Compute Better Spent: Replacing Dense Layers with Structured Matrices (ICML 2024 · similarity 0.86)
- Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices (NeurIPS 2024 · similarity 0.86)
- A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks (NeurIPS 2024 · similarity 0.85)
- Group and Shuffle: Efficient Structured Orthogonal Parametrization (NeurIPS 2024 · similarity 0.85)
- Investigating the Overlooked Hessian Structure: From CNNs to LLMs (ICML 2025 · similarity 0.85)
- BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.