Research map / Architectures and efficiency / Pruning and efficient tuning
Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that heavy fine-tuning of CLIP distorts intra-class feature distributions and hurts tail-class accuracy in long-tail learning, and proposes LIFT, a single-stage lightweight adapter-based fine-tuning method with semantic-aware initialization and test-time ensembling.
Problem
Fine-tuning foundation models for long-tail recognition is common, but how fine-tuning affects tail-class performance was not quantified; heavy full fine-tuning appears to degrade tail classes.
Approach
Full fine-tuning distorts intra-class distributions, breaking the class-conditional consistency assumption underlying logit-adjusted loss; constraining fine-tuning to a few parameters (e.g., AdaptFormer adapters) preserves these distributions while improving separability, and a single-stage pipeline with text-based classifier initialization and test-time ensembling yields accurate, cheap long-tail
Key results
- LIFT with TTE reaches 78.3% overall on ImageNet-LT with 0.62M learnable parameters and 10 epochs, versus 75.7% for BALLAD (149.62M params, 50+10 epochs).
- On Places-LT, LIFT with TTE reaches 52.2% overall (0.18M params, 10 epochs), outperforming LPT's 50.1%; tail accuracy improves from 46.9 to 50.5 without TTE.
- On iNaturalist 2018, LIFT with TTE reaches 80.4% overall (4.75M params, 20 epochs), versus 76.1% for LPT (80+80 epochs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace full fine-tuning with lightweight fine-tuning | CLIP for long-tail learning · ImageNet-LT | tail accuracy: 53.8 -> 72.2 | Helped |
| Replace full fine-tuning with lightweight fine-tuning | CLIP for long-tail learning · Places-LT | tail accuracy: 36.2 -> 51.1 | Helped |
| Add structured lightweight fine-tuning (SLF) (ablation) | LIFT baseline cosine classifier with CE loss · ImageNet-LT | overall accuracy: 60.9 -> 68.9 | Helped |
| Add structured lightweight fine-tuning (SLF) (ablation) | LIFT baseline cosine classifier with CE loss · Places-LT | overall accuracy: 34.3 -> 38.9 | Helped |
| Add logit-adjusted loss (LA) (ablation) | LIFT with SLF · ImageNet-LT | overall accuracy: 68.9 -> 74.9 | Helped |
| Add logit-adjusted loss (LA) (ablation) | LIFT with SLF · Places-LT | overall accuracy: 38.9 -> 48.7 | Helped |
| Add semantic-aware initialization (SAI) (ablation) | LIFT with SLF and LA · ImageNet-LT | tail accuracy: 61.7 -> 71.5 | Helped |
| Add semantic-aware initialization (SAI) (ablation) | LIFT with SLF and LA · Places-LT | tail accuracy: 40.5 -> 50.5 | Helped |
| Add test-time ensembling (TTE) (ablation) | LIFT with SLF, LA, and SAI · ImageNet-LT | overall accuracy: 77.0 -> 78.3 | Helped |
| Add test-time ensembling (TTE) (ablation) | LIFT with SLF, LA, and SAI · Places-LT | overall accuracy: 51.5 -> 52.2 | Helped |
| Add test-time ensembling (TTE) | LIFT · iNaturalist 2018 | overall accuracy: 79.1 -> 80.4 | Helped |
| Use full fine-tuning instead of classifier fine-tuning | CLIP · ImageNet-LT | overall accuracy: 73.5 -> 72.9 | Hurt |
Surprising findings
- Full fine-tuning of CLIP lowers tail-class accuracy on ImageNet-LT (53.8%) compared with classifier-only fine-tuning (68.0%), even with a balanced LA loss.
- Arbitrary (randomly selected) sparse parameter fine-tuning of only 0.1%-2% of weights already outperforms full fine-tuning.
Limitations
- Results rely on CLIP-style foundation models; generalization to other backbones is only partially explored in the supplement.
- The theoretical argument relies on an assumption of class-conditional inconsistency that is supported mainly by empirical observations.
- Heavy fine-tuning hyperparameter sensitivity was examined only with a limited learning-rate grid.
Experimental setup
Datasets: ImageNet-LT Places-LT iNaturalist 2018 CIFAR-100-LT
Benchmarks: ImageNet-LT Places-LT iNaturalist 2018 CIFAR-100-LT
Metrics: overall accuracy head accuracy medium accuracy tail accuracy
Baselines: BALLAD LPT VL-LTR RAC LiVT Decoder GML MiSLAS PaCo NCL BCL RIDE
Builds on: CLIP logit-adjusted loss AdaptFormer VPT LoRA
Closest papers by meaning
- Fine-Tuning is Fine, if Calibrated (NeurIPS 2024 · similarity 0.87)
- Rethinking the Bias of Foundation Model under Long-tailed Distribution (ICML 2025 · similarity 0.86)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.85)
- Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning (ICML 2024 · similarity 0.85)
- LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups (NeurIPS 2025 · similarity 0.85)
- Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.