Research map / Architectures and efficiency / CNN design
Understanding the Covariance Structure of Convolutional Filters
Asher Trockman, Devin Willmott, J Zico Kolter
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| covariance-construction multivariate init (vs uniform) for thawed filters | ConvMixer-512/24 and deeper ConvMixers, ImageNet 50 epochs · ImageNet-1k 50 epochs | accuracy: +0.4 to +3.8 | Improved |
| covariance-construction init vs uniform init for frozen filters | ConvMixer and ConvNeXt-Tiny, ImageNet 50 epochs · ImageNet-1k 50 epochs | accuracy: +2 to +11 points over frozen uniform | Improved |
| covariance-construction init vs uniform init for thawed filters | ConvNeXt-Atto, 50 epochs ImageNet · ImageNet-1k 50 epochs | accuracy: 69.96 to 67.84 (best setting) | Hurt |
| covariance-construction init vs uniform init, thawed filters, 150-epoch training | ConvMixer-1536/24 and ConvNeXt-Tiny · ImageNet-1k 150 epochs | accuracy: ~+0.1 each | Improved |
| frozen-filter covariance init vs uniform thawed init | ConvMixer-1024/32, ImageNet 50 epochs · ImageNet-1k 50 epochs | accuracy: 64.9 frozen uniform to 73.1 frozen ours (vs 72.18 thawed uniform) | Improved |
| covariance-construction init for thawed filters on CIFAR-10 (2x2 patch ConvMixer-256/8) | ConvMixer-256/8 filter size 7 and 9 · CIFAR-10 200 epochs | accuracy: up to +1.1 vs uniform | Improved |
| covariance-construction init for 3x3 filters | ConvMixer and ConvNeXt on CIFAR-10 · CIFAR-10 | accuracy: e.g. 89.69 uniform vs 89.64 ours at 20 epochs (3x3, patch 2) | Hurt |
| frozen filters with covariance-construction init (vs thawed uniform) | ConvMixer-256/24 patch 2, large filters · CIFAR-10 20 epochs | accuracy: up to 5.0% margin over thawed uniform | Improved |
| covariance transfer from narrower reference model (stats from 1/8 width) vs same-size reference (ablation) | ConvMixer-256/8 covariance transfer · CIFAR-10 | accuracy: mixed magnitude | Improved |
| Kronecker-factorized covariance (A kron A) instead of full covariance for transfer (ablation) | covariance transfer ConvMixer · CIFAR-10 | accuracy: slight | Improved |
| ImageNet 10-epoch: our init vs uniform (thawed) for ConvMixer-512/12 patch 14 (ablation) | ConvMixer-512/12 10-epoch ImageNet · ImageNet-1k 10 epochs | accuracy: 54.5 to 55.5 | Improved |
Closest papers by meaning
- Exploiting Invariance in Training Deep Neural Networks (AAAI 2022 · similarity 0.89)
- CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters (CVPR 2022 · similarity 0.88)
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.88)
- Learning Invariances in Neural Networks from Training Data (NeurIPS 2020 · similarity 0.88)
- Is Second-Order Information Helpful for Large-Scale Visual Recognition? (ICCV 2017 · similarity 0.88)
- Unveiling the Unseen: Identifiable Clusters in Trained Depthwise Convolutional Kernels (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
