Research map / Architectures and efficiency / Pruning and efficient tuning
Large Convolutional Model Tuning via Filter Subspace
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes parameter-efficient fine-tuning of pre-trained ConvNets (and Stable Diffusion) by updating only spatial filter atoms of a sparse filter-subspace decomposition while freezing atom coefficients, with recursive overcomplete atoms for more capacity.
Problem
Full fine-tuning and LoRA of large pre-trained models are parameter-heavy and overfit on small downstream datasets, degrading the generalization of pre-trained models (e.g., diversity in generative models).
Approach
Decompose each convolution filter as a linear combination of filter atoms (spatial-only) with fixed atom coefficients (channel mixing), fine-tune only the atoms, and expand tunable capacity by recursively decomposing atoms over another overcomplete atom set with intra-channel coefficients beta; the scheme extends to linear layers via Kronecker-style decomposition.
Key results
- On VTAB-1k with ConvNeXt-B (ImageNet-21K), the method reaches 73.59 mean accuracy with 0.45M tunable parameters versus 67.53 for LoRA (17.4M).
- For few-shot Stable Diffusion generation, the method attains the lowest VTAB FID mean of 54.7 with 1.11M parameters, versus 57.5 for LoRA with 22.67M and 61.3 for full fine-tuning (860M).
- On the ResNet50/CIFAR-100 validation study, fine-tuning only filter atoms (about 0.21M parameters) improves accuracy from 55.4 (linear probe) to 66.86 and the overcomplete variant reaches 78.7 to 81.8 with more parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune filter atoms and linear-layer atoms instead of full fine-tuning | ConvNeXt-B · VTAB-1k | accuracy: 72.14 -> 73.59 | Helped |
| Fine-tune filter atoms and linear-layer atoms instead of LoRA updates | ConvNeXt-B · VTAB-1k | accuracy: 67.53 -> 73.59 | Helped |
| Fine-tune filter atoms and linear-layer atoms instead of LoRA updates | Stable Diffusion · VTAB image generation | Mean FID: 57.5 -> 54.7 | Helped |
| Fine-tune filter atoms and linear-layer atoms instead of full fine-tuning | ConvNeXt-B · CIFAR-100 | accuracy: 94.1 -> 91.8 | Hurt |
| Fine-tune filter atoms and linear-layer atoms instead of full fine-tuning | ConvNeXt-B · ImageNet-1k | accuracy: 85.8 -> 84.9 | Hurt |
| Add overcomplete filter atoms and their coefficients β to filter-atom tuning (ablation) | ResNet50 · CIFAR-100 | accuracy: 66.86 -> 78.7 | Helped |
| Add linear-layer atoms D_c to filter-atom tuning (ablation) | ResNet50 · CIFAR-100 | accuracy: 66.86 -> 75.03 | Helped |
| Increase filter atom count from m=6 to m=12 (ablation) | ResNet50 · CIFAR-100 | accuracy: 66.86% -> 68.68% | Helped |
| Increase overcomplete atom count m₁ from 3 to 5 (ablation) | ResNet50 · CIFAR-100 | accuracy: 78.69 -> 78.9 | Helped |
| Fine-tune atom coefficients α in addition to D and D_c (ablation) | Stable Diffusion · Customized concept generation | Diversity: 20.42 -> 3.17 | Hurt |
| Fine-tune atom coefficients α in addition to D and D_c (ablation) | Stable Diffusion · Customized concept generation | T2I Alignment: 0.301 -> 0.201 | Hurt |
| Fine-tune D and D_c rather than only D (ablation) | ResNet50 · CIFAR-100 | accuracy: 66.86 -> 75.03 | Helped |
Surprising findings
- Fine-tuning the atom coefficients alpha (config C4) yields high fidelity but collapses diversity and text-image alignment, indicating overfitting.
- On full-dataset CIFAR-100 and ImageNet-1K fine-tuning, full fine-tuning beats all parameter-efficient methods, which the authors attribute to ample data preventing overfitting.
Limitations
- Method is mainly designed for convolutional networks; its extension to linear layers is under-explored.
- Increasing the number of atoms does not easily yield further accuracy gains; more accuracy needs the overcomplete set at extra parameter cost.
Experimental setup
Datasets: VTAB-1k CIFAR-100 ImageNet-1K Dreambooth CustomConcept101 LAION CUB
Benchmarks: VTAB-1k CIFAR-100 ImageNet-1K Dreambooth concepts
Metrics: accuracy FID fidelity (DINOv2 cosine similarity) diversity (Vendi score) text-image alignment (CLIP cosine similarity) parameter count
Baselines: Full fine-tuning LoRA LoHa LoKr BitFit DiffFit OFT COFT Linear probing Adapter FacT SSF
Builds on: LoRA sparse coding Kronecker decomposition fine-tuning DiffFit BitFit
Closest papers by meaning
- Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models (CVPR 2025 · similarity 0.90)
- PROFIT: A Specialized Optimizer for Deep Fine Tuning (NeurIPS 2025 · similarity 0.84)
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.84)
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.84)
- LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views (ICML 2024 · similarity 0.84)
- SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.