Research map / Label-efficient and robust learning / Recognition and long-tail
Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domains
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLAdapter, a cluster-attention adapter with learnable cluster centers and per-cluster transformation matrices plus a two-stage staged fine-tuning (SFT) strategy, evaluated across 10 data-limited scientific and OOD/3D vision benchmarks.
Problem
Adapting large-scale pre-trained vision foundation models to data-limited, heterogeneous scientific downstream tasks, including OOD scenarios, where full fine-tuning and existing PEFT methods (VPT, L2-SP, LoRA) are unstable or underperform.
Approach
Learnable cluster centers compute cosine-similarity attention over pre-trained token features, which weights a set of learnable D×D transformation matrices applied per sample; a unified interface makes it work for CNN/ViT/2D/3D backbones, and a staged schedule first trains only the adapter and head (LP) before full fine-tuning.
Key results
- On InsPLAD-fault, CLAdapter (ConvNeXt-B) reaches 98.29 average ROC, above the best baseline AttentDifferNet at 94.34.
- On BreakHis, CLAdapter (ViT-B) reaches 93.71 F1 after two SFT stages, with the authors reporting gains of 7.97% and 10.02% F1 over the best baseline for ConvNeXt-B and ViT-B.
- On PACS (OOD), CLAdapter reaches 91.41% accuracy, versus 87.79% for full fine-tuning and 76.76% for VPT-Adapter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CLAdapter instead of the strongest listed baseline | ViT-L · Tiny-ImageNet | accuracy: 86.43% to 94.21% | Helped |
| apply CLAdapter instead of full fine-tuning | ViT-B · PACS | accuracy: by 3.62% | Helped |
| increase cluster centers to K=20 (ablation) | CLAdapter ViT-B · BreakHis | F1: 90.40 -> 93.71; 90.73 -> 93.71; 90.41 -> 93.71; 92.59 -> 93.71; 92.14 -> 93.71; 91.77 -> 93.71; 89.03 -> 93.71; 90.42 -> | Helped |
| use SFT stage 2 rather than stage 1 only (ablation) | CLAdapter ConvNeXt-B · BreakHis | F1: 80.89 -> 91.66 | Helped |
| use SFT stage 2 rather than stage 1 only (ablation) | CLAdapter ViT-B · BreakHis | F1: 84.34 -> 93.71 | Helped |
| use LAION-2B rather than ImageNet-21K pre-training (ablation) | CLAdapter ViT-B · PACS | accuracy: 91.41 -> 97.62 | Helped |
| use LAION-2B rather than ImageNet-21K pre-training (ablation) | CLAdapter ConvNeXt-B with SFT stage 2 · BreakHis | F1: 88.60 -> 91.66 | Helped |
| use LAION-2B rather than ImageNet-21K pre-training (ablation) | CLAdapter ViT-B with SFT stage 2 · BreakHis | F1: 90.19 -> 93.71 | Helped |
| use ImageNet-21K rather than LAION-400M or LAION-2B pre-training (ablation) | CLAdapter ConvNeXt-B with SFT stage 1 · BreakHis | F1: 71.72 -> 80.89; 75.17 -> 80.89 | Mixed |
| use ImageNet-21K rather than LAION-400M or LAION-2B pre-training (ablation) | CLAdapter ViT-B with SFT stage 1 · BreakHis | F1: 80.63 -> 84.34; 80.67 -> 84.34 | Mixed |
Surprising findings
- Full fine-tuning and L2-SP collapse on PACS when pre-trained on LAION-2B (47.17% and 45.56%), which the authors attribute to pattern collapse.
- With only the first SFT stage (frozen backbone), ImageNet-21K pre-training beats LAION-400M and LAION-2B on BreakHis.
Limitations
- Not designed or validated for detection or segmentation, as the authors state.
- Cluster count K is tuned on BreakHis only and is fixed at 20 for all experiments.
- Reported gains on several datasets rely on small test splits and inconsistent per-dataset baselines and backbones.
Experimental setup
Datasets: ImageNet-21K LAION-400M LAION-2B Kinetics-400 Tiny-ImageNet PACS BreakHis HCRF Apple Foliar Disease WHU-RS19 KTH-TIPS-2b InsPLAD-fault
Benchmarks: Tiny-ImageNet PACS BreakHis HCRF Apple Foliar Disease WHU-RS19 KTH-TIPS-2b InsPLAD-fault UCF101 HMDB51
Metrics: Top-1 accuracy precision recall F1 AUROC
Baselines: VPT L2-SP LoRA DoRA MoRA Linear probing Full fine-tuning SFT GasHis-Transformer AttentDifferNet VideoMAE Swin-L
Builds on: Visual Prompt Tuning L2-SP LoRA Vision Transformer ConvNeXt
Closest papers by meaning
- Transferring Knowledge From Large Foundation Models to Small Downstream Models (ICML 2024 · similarity 0.85)
- Sparse autoencoders reveal selective remapping of visual concepts during adaptation (ICLR 2025 · similarity 0.84)
- Enhancing Foundation Models with Federated Domain Knowledge Infusion (ICML 2025 · similarity 0.84)
- Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · similarity 0.84)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.84)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.