Research map / Label-efficient and robust learning / Recognition and long-tail
Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Tuning Stable Rank Shrinkage (TSRS), a noise-sensitivity regularizer that approximates stable rank to reduce structural risk when fine-tuning pre-trained CNNs and ViTs on small target datasets, and adds it to existing fine-tuning methods.
Problem
Fine-tuned models retain high model complexity inherited from large-scale pre-training, which raises structural risk and hurts generalization when target data is small; existing fine-tuning methods only re-weight source knowledge.
Approach
Estimate the stable rank of the whole network via the noise sensitivity of block features under input Gaussian noise, and add the summed normalized noise-sensitivity loss over blocks as a regularizer so that fine-tuning lowers model complexity; it plugs into BSS, DELTA, Co-Tuning, and UOTS.
Key results
- On ResNet50 with Cars at 15% data, BSS+TSRS reaches 47.22% vs 44.00% for BSS, and Co-Tuning+TSRS improves to 51.98% vs 47.94%.
- TSRS improves ViT-B Baseline on CUB at 15% from 64.96% to 70.76% top-1.
- On TransUNet, TSRS raises Synapse multi-organ DSC from 77.16 to 78.30 and reduces HD from 33.17 to 29.28.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TSRS to vanilla fine-tuning | ResNet50 · CUB 15% training | Top-1 accuracy (%): 45.25±0.12 -> 52.76±0.21 | Helped |
| Add TSRS to BSS | ResNet50 · Cars 15% training | Top-1 accuracy (%): 44.00±0.32 -> 47.22±0.23 | Helped |
| Add TSRS to vanilla fine-tuning | ViT-B · Aircraft 15% training | Top-1 accuracy (%): 38.67±0.08 -> 39.24±0.03 | Helped |
| Add TSRS to vanilla fine-tuning | TransUNet · Synapse multi-organ | DSC: 77.16 -> 78.30 | Helped |
| Add TSRS to vanilla fine-tuning | DETR · LIDC 15% training | AP: 53.26 -> 57.02 | Helped |
| Use TSRS rather than L1 regularization | ResNet50 with BSS · Cars 15% training | Top-1 Accuracy(%): 42.96 -> 46.30 | Helped |
| Use TSRS rather than L2 regularization | ResNet50 with BSS · Cars 15% training | Top-1 Accuracy(%): 44.00 -> 46.30 | Helped |
| Increase model size from ResNet18 to ResNet50 while using BSS+TSRS (ablation) | BSS+TSRS · Cars 15% training | Top-1 Accuracy(%): 38.73±0.17 -> 47.22±0.23 | Helped |
| Increase model size from ResNet50 to ResNet152 while using BSS+TSRS (ablation) | BSS+TSRS · Cars 15% training | Top-1 Accuracy(%): 47.22±0.23 -> 51.12±0.07 | Helped |
| Increase noise intensity ϕ (ablation) | TSRS fine-tuning · CUB 15% training | validation performance | Mixed |
| Vary weighting hyper-parameter α (ablation) | TSRS fine-tuning · CUB 15% training | validation performance | Mixed |
| Apply the TSRS constraint starting in shallow blocks (ablation) | TSRS fine-tuning · CUB | validation performance | Hurt |
Surprising findings
- Noise sensitivity of the DELTA-fine-tuned model was even higher than before fine-tuning, attributed to retained source-network behavior.
Limitations
- May yield little improvement when source and target data volumes are similar.
- Adds an extra forward pass during training, increasing training cost.
Experimental setup
Datasets: ImageNet-1k CUB-200-2011 Stanford Cars FGVC Aircraft Synapse multi-organ ISIC2018 MRSpineSeg LIDC MNIST
Benchmarks: CUB-200-2011 Stanford Cars FGVC Aircraft Synapse multi-organ ISIC2018 MRSpineSeg LIDC
Metrics: top-1 accuracy DSC HD AP Lipschitz constant Hessian eigenvalues noise sensitivity
Baselines: Baseline fine-tuning DELTA BSS Co-Tuning UOTS L1 L2 SRN Dynamic ART ANO
Builds on: BSS DELTA Co-Tuning UOTS Arora et al. noise sensitivity and stable rank
Closest papers by meaning
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.89)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.87)
- RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.87)
- Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning (ICLR 2025 · similarity 0.87)
- Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods (ICML 2025 · similarity 0.87)
- PROFIT: A Specialized Optimizer for Deep Fine Tuning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.