AI Paper Atlas

Research map / Architectures and efficiency / CNN design

Understanding the Covariance Structure of Convolutional Filters

Asher Trockman, Devin Willmott, J Zico Kolter

ICLR 2023 · Architectures and efficiency › CNN design · approach: Theory of convolutional inductive bias · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
covariance-construction multivariate init (vs uniform) for thawed filtersConvMixer-512/24 and deeper ConvMixers, ImageNet 50 epochs · ImageNet-1k 50 epochsaccuracy: +0.4 to +3.8Improved
covariance-construction init vs uniform init for frozen filtersConvMixer and ConvNeXt-Tiny, ImageNet 50 epochs · ImageNet-1k 50 epochsaccuracy: +2 to +11 points over frozen uniformImproved
covariance-construction init vs uniform init for thawed filtersConvNeXt-Atto, 50 epochs ImageNet · ImageNet-1k 50 epochsaccuracy: 69.96 to 67.84 (best setting)Hurt
covariance-construction init vs uniform init, thawed filters, 150-epoch trainingConvMixer-1536/24 and ConvNeXt-Tiny · ImageNet-1k 150 epochsaccuracy: ~+0.1 eachImproved
frozen-filter covariance init vs uniform thawed initConvMixer-1024/32, ImageNet 50 epochs · ImageNet-1k 50 epochsaccuracy: 64.9 frozen uniform to 73.1 frozen ours (vs 72.18 thawed uniform)Improved
covariance-construction init for thawed filters on CIFAR-10 (2x2 patch ConvMixer-256/8)ConvMixer-256/8 filter size 7 and 9 · CIFAR-10 200 epochsaccuracy: up to +1.1 vs uniformImproved
covariance-construction init for 3x3 filtersConvMixer and ConvNeXt on CIFAR-10 · CIFAR-10accuracy: e.g. 89.69 uniform vs 89.64 ours at 20 epochs (3x3, patch 2)Hurt
frozen filters with covariance-construction init (vs thawed uniform)ConvMixer-256/24 patch 2, large filters · CIFAR-10 20 epochsaccuracy: up to 5.0% margin over thawed uniformImproved
covariance transfer from narrower reference model (stats from 1/8 width) vs same-size reference (ablation)ConvMixer-256/8 covariance transfer · CIFAR-10accuracy: mixed magnitudeImproved
Kronecker-factorized covariance (A kron A) instead of full covariance for transfer (ablation)covariance transfer ConvMixer · CIFAR-10accuracy: slightImproved
ImageNet 10-epoch: our init vs uniform (thawed) for ConvMixer-512/12 patch 14 (ablation)ConvMixer-512/12 10-epoch ImageNet · ImageNet-1k 10 epochsaccuracy: 54.5 to 55.5Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy