Research map / Architectures and efficiency / Pruning and efficient tuning
Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines functionally equivalent features via natural isomorphisms in a category-theory view, proves LMC implies such equivalence, and proposes Iterative Feature Merging (IFM) to measure feature complexity and prune networks without fine-tuning.
Problem
Feature similarity measures are vague because exactly equivalent features rarely exist, so they cannot say which features are redundant or mergeable in a trained network.
Approach
Defines functionally equivalent features as those related by a natural isomorphism between functors (network parameterizations) from a category of network structure, shows LMC implies such equivalence, defines feature complexity as the width of the poset of non-equivalent features, and measures it by iteratively merging weight-matched features.
Key results
- On CIFAR10 VGG16, IFM reduces parameters to 23.03% while keeping test accuracy at 91.78% (vs 93.51% original), without fine-tuning.
- On CIFAR10 ResNet18, up to 60% of parameters can be reduced without significantly affecting test accuracy.
- On ImageNet, IFM reduces only about 5% of parameters for VGG and about 10% for ResNet50, and is less effective on the larger dataset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Merge functionally equivalent features with IFM | VGG16 · CIFAR10 | testing accuracy: 93.51% -> 91.78% | Hurt |
| Merge functionally equivalent features with IFM | VGG16 · CIFAR10 | remaining parameters: 23.03% remaining | Helped |
| Merge more features | VGG networks · ImageNet | parameters reduced: approximately 5% | Helped |
| Merge more features | ResNet50 · ImageNet | parameters reduced: approximately 10% | Helped |
| Use ResNet50_v2 instead of ResNet50_v1 | ResNet50 · ImageNet | top-1 accuracy: 76.13% -> 80.86% | Helped |
| Merge more features with IFM | ResNet50_v2 · ImageNet | testing accuracy | Hurt |
| Use IFM instead of INN | ResNet18 · CIFAR10 | accuracy | Helped |
| Increase network size | VGG and ResNet models · CIFAR10 | mergeable equivalent features | Helped |
| Increase task difficulty from CIFAR10 to ImageNet | VGG and ResNet models · ImageNet | mergeable parameters | Hurt |
Surprising findings
- On ImageNet, IFM is much less effective than on CIFAR10, with only about 5-10% parameter reduction.
- ResNet18 outperforms ResNet34 and ResNet50 on CIFAR10 in the authors' experiments.
Limitations
- Only image classification experiments with VGG and ResNet; not validated on other architectures or tasks.
- Limited parameter reduction on large-scale ImageNet, where IFM is less effective.
- Feature merging is approximate; the natural-isomorphism theory uses permutation-based correspondence and the merge rule is heuristic.
Experimental setup
Datasets: CIFAR10 ImageNet
Benchmarks: CIFAR10 ImageNet
Metrics: top-1 test accuracy percentage of remaining parameters time per iteration
Baselines: INN
Builds on: linear mode connectivity weight matching layerwise linear feature connectivity
Closest papers by meaning
- Understanding Visual Feature Reliance through the Lens of Complexity (NeurIPS 2024 · similarity 0.87)
- Exploring the Complexity of Deep Neural Networks through Functional Equivalence (ICML 2024 · similarity 0.86)
- Complexity Matters: Feature Learning in the Presence of Spurious Correlations (ICML 2024 · similarity 0.86)
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · similarity 0.86)
- ReSi: A Comprehensive Benchmark for Representational Similarity Measures (ICLR 2025 · similarity 0.86)
- Explainable Neural Networks with Guarantee: A Sparse Estimation Approach (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.