Research map / Learning theory and algorithms / Matrix and tensor methods
Weight matrices compression based on PDB model in deep neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Population Double Bulk (PDB) model and PDBLS estimator from random matrix theory to find a noise-information boundary in trained weight matrices, then compresses weights by removing the lower bulk, reducing rank while preserving or improving test accuracy.
Problem
Compressing trained DNN weight matrices by filtering noisy eigenvalues requires locating the noise-information boundary; existing PUB-based methods assume a single homogeneous bulk variance, which fits eigenvalue distributions poorly and yields overly aggressive compression.
Approach
Generalize the PUB model to a two-bulk PDB model with heterogeneous bulk variances, estimate its parameters with a least-squares procedure (PDBLS) grounded in RMT, and treat the boundary between bulk1 and bulk2 as the noise cutoff, zeroing only bulk2 and replacing spikes with recovered population spikes.
Key results
- PDB-based compression raises T5-base RTE test accuracy from 0.7174 to 0.7536 and averages 0.7891 vs. 0.7783 base across architectures, outperforming PUB (0.7827).
- PDB model matches empirical spectral moments better than PUB; e.g., ResNet18 ImageNet third moment empirical 402.79 vs PDBLS 377.50 vs BEMA 126.79.
- On FCNN MNIST, the PDB boundary keeps 55/350 rank (15.72%) while PUB keeps 30/350 (8.57%), with accuracy 0.9804 vs 0.9791.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply PDB noise-filtering compression | FCNN · MNIST | test accuracy: 0.9799 -> 0.9804 | Helped |
| apply PDB noise-filtering compression | ResNet18 · CIFAR10 | test accuracy: 0.8349 -> 0.8384 | Helped |
| apply PDB noise-filtering compression | VGG16 · CIFAR10 | test accuracy: 0.8418 -> 0.8422 | Helped |
| apply PDB noise-filtering compression | BERT · RTE | test accuracy: 0.7029 -> 0.7319 | Helped |
| apply PDB noise-filtering compression | BERT · SciTail | test accuracy: 0.9055 -> 0.9155 | Helped |
| apply PDB noise-filtering compression | T5-base · RTE | test accuracy: 0.7174 -> 0.7536 | Helped |
| apply PDB noise-filtering compression | T5-base · SciTail | test accuracy: 0.9025 -> 0.9243 | Helped |
| apply PDB noise-filtering compression | ViT-L · DTD | test accuracy: 0.7452 -> 0.7533 | Helped |
| apply PDB noise-filtering compression | ViT-L · SUN397 | test accuracy: 0.7680 -> 0.7771 | Helped |
Surprising findings
- Compression sometimes improves test accuracy over the uncompressed baseline, notably with 60% label noise.
- The empirical 0.55 compression ratio from prior work does not fully compress the weight matrix in the authors' experiments.
Limitations
- Only two bulks are modeled; more general multi-bulk models (M>=3) were not adopted.
- Rank reductions are modest for some settings (e.g., 13.5% to 35% of rank retained), and the gains in accuracy are small in most clean-label settings.
- Hyperparameters such as the number of least-squares points and the initial spike estimate (K0) are set by hand.
Experimental setup
Datasets: MNIST CIFAR10 ImageNet RTE SciTail DTD SUN397
Benchmarks: MNIST CIFAR10 ImageNet RTE SciTail DTD SUN397
Metrics: test accuracy rank ratio after compression spectral moment accuracy computation time
Baselines: BEMA Kernel SLR Naive SVD PUB model
Builds on: Population Unit Bulk (PUB) model Bulk Eigenvalue Matching Analysis (BEMA) Kernel estimation of spectral density Naive SVD compression Marchenko-Pastur law
Closest papers by meaning
- Double Viterbi: Weight Encoding for High Compression Ratio and Fast On-Chip Reconstruction for Deep Neural Network (ICLR 2019 · similarity 0.85)
- Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters (ICLR 2019 · similarity 0.85)
- On Compressing Deep Models by Low Rank and Sparse Decomposition (CVPR 2017 · similarity 0.85)
- Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization (ICLR 2025 · similarity 0.83)
- Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networks (ICLR 2024 · similarity 0.83)
- Debiased Distribution Compression (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.