Research map / Label-efficient and robust learning / Recognition and long-tail
Rethinking the Bias of Foundation Model under Long-tailed Distribution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes how pre-training (parameter) imbalance in foundation models like CLIP propagates to long-tailed downstream fine-tuning, and proposes a causal backdoor-adjustment method that fuses predictions from multiple foundation models to treat incomplete semantic factors as confounders.
Problem
Long-tailed downstream learning with fine-tuned foundation models ignores the bias inherited from imbalanced pre-training data (parameter imbalance), which standard re-balancing such as logit adjustment cannot fix.
Approach
Treats incomplete semantic factors captured by differently pre-trained foundation models (CLIP, OpenCLIP, MetaCLIP) as confounders and applies a backdoor adjustment, approximating the sum over confounder values by fusing LA-adjusted predictions from these models with PEFT (AdaptFormer/VPT) fine-tuning.
Key results
- On Places365-LT the method reaches 53.01% overall accuracy, versus 51.5% for LIFT and 50.1% for VL-LTR, using 0.54M trainable parameters for 10 epochs.
- On ImageNet-LT the method reaches 79.57% overall, improving on LIFT (77.0%) by about 2.6 points; on iNaturalist2018 it reaches 82.01% overall, versus 80.2% for RAC.
- Ablation shows that increasing the number of foundation models M from 1 to 3 improves ImageNet-LT D-Few accuracy for OpenCLIP by about 3.3 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply the proposed backdoor adjustment instead of LIFT | ViT-B/16 fine-tuning · Places365-LT | All accuracy: 51.5 -> 53.01 | Helped |
| apply the proposed backdoor adjustment instead of GML | ViT-B/16 fine-tuning · ImageNet-LT | All accuracy: 78.0 -> 79.57 | Helped |
| apply the proposed backdoor adjustment instead of RAC | ViT-B/16 fine-tuning · iNaturalist2018 | All accuracy: 80.2 -> 82.01 | Helped |
| use CLIP, OpenCLIP, and MetaCLIP instead of only OpenCLIP as semantic factors (ablation) | backdoor adjustment with AdaptFormer · ImageNet-LT | D-Few accuracy: 71.70 -> 74.99 | Helped |
| use CLIP, OpenCLIP, and MetaCLIP instead of only OpenCLIP as semantic factors (ablation) | backdoor adjustment with VPT · ImageNet-LT | D-Few accuracy: 70.91 -> 74.16 | Helped |
| use CLIP, OpenCLIP, and MetaCLIP instead of only OpenCLIP as semantic factors (ablation) | backdoor adjustment with VPT · Places365-LT | D-Few accuracy: 50.68 -> 51.34 | Helped |
| extend GLA into the training loss (GLA-Train) instead of using GLA | CLIP fine-tuning · ImageNet-LT | Overall accuracy: 77.42 -> 77.10 | Hurt |
Surprising findings
- Logit adjustment during training does not alleviate parameter imbalance; tail classes grouped by parameter imbalance (P-Few) can deteriorate under LA.
- Classes that are both data-tail and parameter-tail perform worst, and a reversed-imbalance setting outperforms the consistent setting in the downsampled Places experiment.
Limitations
- Using M foundation models multiplies inference compute roughly linearly (FLOPs 239 to 717 for M=1 to 3).
- Parameter imbalance is estimated via GLA on the validation set, not from the true pre-training distribution, which is inaccessible.
- Results are reported only for image classification on three long-tailed benchmarks, with extension to detection and segmentation left as future work.
Experimental setup
Datasets: ImageNet-LT Places365-LT iNaturalist2018
Benchmarks: ImageNet-LT Places365-LT iNaturalist2018
Metrics: top-1 accuracy D-Many/D-Medium/D-Few accuracy
Baselines: LiVT LPT VL-LTR RAC BALLAD Decoder LIFT GML
Builds on: Logit Adjustment Generalized Logit Adjustment AdaptFormer Visual Prompt Tuning LIFT
Closest papers by meaning
- LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups (NeurIPS 2025 · similarity 0.87)
- Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts (ICML 2024 · similarity 0.86)
- Reducing Balancing Error for Causal Inference via Optimal Transport (ICML 2024 · similarity 0.86)
- Long-tailed Recognition with Model Rebalancing (NeurIPS 2025 · similarity 0.86)
- Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation (NeurIPS 2025 · similarity 0.85)
- Long-Tailed Learning as Multi-Objective Optimization (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.