Research map / Architectures and efficiency / CNN design
Learning Compressed Transforms with Low Displacement Rank
Anna Thomas, Albert Gu, Tri Dao, Atri Rudra, Christopher Ré
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn displacement operators rather than fixing them to the Toeplitz-like operators | LDR-SD weight layers · MNIST-bg-rot, MNIST-noise, CIFAR-10, and NORB | test accuracy: 3.14, 2.70, 3.55, and 3.37 accuracy points | Improved |
| Use learned operators instead of fixed operators at rank one | Single hidden layer network · MNIST-bg-rot, MNIST-noise, and NORB | test accuracy: 1.73, 13.30, and 2.92 accuracy points | Improved |
| Increase LDR-SD displacement rank to 16 (ablation) | LDR-SD fully-connected layers · MNIST-bg-rot, MNIST-noise, CIFAR-10, and NORB | test accuracy: 5.87, 15.05, 0.74, and 6.86 accuracy points | Improved |
| Learn subdiagonal operators instead of using fixed Toeplitz-like operators | LSTM input-hidden weights · WikiText-2 | test perplexity: 19.94-42.92 perplexity points | Improved |
| Replace convolutional channels and pooling with two generic LDR-SD matrices | CIFAR-10 CNN · CIFAR-10 | test accuracy: within 1.05 accuracy points | Hurt |
| Use learned LDR-SD channels rather than fixed Toeplitz-like channels at rank 16 | Two structured-matrix layers · CIFAR-10 | test accuracy: 57.29 -> 59.36 | Improved |
| Increase the Toeplitz-like rank from 1 to 2 rather than broaden the operator class to LDR-SD at rank 1 (ablation) | Compressed fully-connected layer · Image classification datasets | test accuracy | Hurt |
| Train the structured LDR-TD model with less training data | Single hidden layer network · MNIST-noise | test accuracy | Improved |
| Train the structured LDR-TD model with less training data | CNN · MNIST-noise | test accuracy | Improved |
| Replace the LSTM input-hidden matrix with LDR-SD rather than an unstructured layer | Single-layer LSTM · WikiText-2 | test perplexity | Hurt |
| Use LDR-SD instead of an unstructured fully-connected layer at rank 1 | Single hidden layer network · CIFAR-10 | test accuracy | Hurt |
Closest papers by meaning
- Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networks (ICLR 2024 · similarity 0.86)
- Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.86)
- From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications (ICML 2025 · similarity 0.85)
- Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation (ICML 2024 · similarity 0.85)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.85)
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
