Research map / Architectures and efficiency / CNN design
Unveiling the Unseen: Identifiable Clusters in Trained Depthwise Convolutional Kernels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that trained depthwise convolution kernels in modern DS-CNNs (ConvNeXt, ConvNeXtV2, HorNet, MogaNet, EfficientNet, etc.) cluster into a few DoG-like patterns (on/off-center, first/second derivatives, crosses) using an autoencoder with a 1D code.
Problem
Depthwise convolution kernels in DS-CNNs are poorly understood and largely unstudied for interpretability, unlike regular convolution kernels which appear uninterpretable beyond early layers.
Approach
An autoencoder with a 1D sigmoid code is trained on over a million centered, unit-norm depthwise kernels; decoding uniformly sampled codes yields a spectrum of prototypes, which are labeled manually and used to assign filters to clusters by minimum cosine dissimilarity threshold, revealing DoG-like classes that are consistent across layers and models.
Key results
- ConvNeXtV2-tiny (ImageNet-22k) had 98.16% of 7x7 depthwise kernels clustered, and ConvNeXt-tiny 95.21%.
- Clusters from ConvNeXtV2 kernels are dominated by on/off-center DoG patterns, followed by cross patterns and first-derivative patterns; second derivatives are least frequent.
- Initializing ConvMixer-768-32 with DoG-like kernels raised clustered filters from 56.65% to 96.03% and top-1 accuracy from 66.35% to 69.01% in a single run.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize ConvMixer filters with DoG patterns instead of Kaiming initialization (ablation) | ConvMixer-768-32 · ImageNet, 50 epochs | accuracy: 66.35 -> 69.01 | Helped |
| Initialize ConvMixer filters with DoG patterns instead of Kaiming initialization (ablation) | ConvMixer-768-32 · 7×7 depthwise filters, 50 epochs | clustered filters: 56.65 -> 96.03 | Helped |
| Train on only 10 ImageNet classes without regularization rather than full ImageNet (ablation) | ConvMixer-768-32 · ImageNet filter patterns | presence of discernible patterns | Hurt |
| Train on CIFAR-10 rather than full ImageNet (ablation) | ConvMixer-768-32 · CIFAR-10 filter patterns | presence of discernible patterns | Hurt |
| Use stricter clustering threshold for 5×5 kernels than for 7×7 kernels (ablation) | Autoencoder-based filter clustering · 5×5 and 7×7 depthwise filters | clustering robustness | Helped |
Surprising findings
- Cross-shaped filters increase in proportion in later layers while DoG and first-derivative filters decrease, with cross motifs dominating the final convolutional layer.
- Overfitted ConvMixer trained on 10 ImageNet classes or on CIFAR-10 lacks clear kernel patterns, suggesting pattern emergence depends on data diversity.
Limitations
- Cluster labels are assigned manually based on visual inspection of the decoded spectrum.
- Autoencoder fits poorly on 3x3 kernels, requiring a different k-means approach; the 1D code trades reconstruction quality for interpretability.
- Cross-shaped filter mathematical form remains unknown; the Gaussian-sum hypothesis is only approximate.
Experimental setup
Datasets: ImageNet-1k ImageNet-21k CIFAR-10
Benchmarks: ImageNet
Metrics: percentage of filters clustered (reconstruction cosine dissi top-1 accuracy
Baselines: k-means clustering regular convolution kernels
Builds on: ConvNeXt ConvNeXtV2 MobileNet
Closest papers by meaning
- The Quest for Universal Master Key Filters in DS-CNNs (NeurIPS 2025 · similarity 0.90)
- The Master Key Filters Hypothesis: Deep Filters Are General (AAAI 2025 · similarity 0.89)
- CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters (CVPR 2022 · similarity 0.89)
- Approximation and Learning with Deep Convolutional Models: a Kernel Perspective (ICLR 2022 · similarity 0.89)
- Rethinking Depthwise Separable Convolutions: How Intra-Kernel Correlations Lead to Improved MobileNets (CVPR 2020 · similarity 0.88)
- PICNN: A Pathway towards Interpretable Convolutional Neural Networks (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.