Research map / Architectures and efficiency / Pruning and efficient tuning
Towards Inadequately Pre-trained Models in Transfer Learning
Andong Deng, Xingjian Li, Di Hu, Tianyang Wang, Haoyi Xiong, Cheng-Zhong Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| stop pre-training early (middle epoch) when using the model as a frozen feature extractor | ResNet-18 CIFAR10 to MNIST · MNIST (1000-sample toy) | FE accuracy: 88.47 -> 96.47 | Improved |
| pre-train fully rather than stopping early when fine-tuning the whole model | ResNet-18 CIFAR10 to MNIST · MNIST (full) | FT accuracy: 99.30 -> 99.46 | Improved |
| longer pre-training on the main spectral component Fm for FE (ablation) | ResNet-18 CIFAR10 to MNIST · MNIST FE | Fm accuracy: 88.24 -> 58.74 | Hurt |
| longer pre-training on residual component Fr for FE (ablation) | ResNet-18 CIFAR10 to MNIST · MNIST FE | Fr accuracy: 55.45 -> 71.28 | Improved |
| use checkpoint selected at 70 epochs instead of fully trained 90 epochs as image feature backbone | PVSE image-text retrieval · MSCOCO | rsum: 132.67 -> 138.66 | Improved |
| use 70-epoch checkpoint for weight-imprinting few-shot classification | ResNet50 on CUB-200-2011 · CUB-200-2011 N=1 | accuracy: 52.11 -> 52.92 | Improved |
| use transferability scores (LogME, LEEP, NCE) to select the best pre-training checkpoint | checkpoint selection · FGVC Aircraft, CIFAR10, Flowers102 | correlation with FT performance | Mixed |
| use self-supervised MoCo-v2 pre-training instead of supervised | feature extractor transfer · selected datasets | FE accuracy vs epoch | Improved |
| train Swin-T longer for feature extraction | Swin-T · CIFAR100, DTD, Flower102, Aircraft, Indoor | FE accuracy: 75.72 (epoch 250) vs 75.58 (epoch 300) on CIFAR100 | No clear effect |
Closest papers by meaning
- Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components (AAAI 2025 · similarity 0.89)
- Do Better ImageNet Models Transfer Better? (CVPR 2019 · similarity 0.89)
- Features are fate: a theory of transfer learning in high-dimensional regression (ICML 2025 · similarity 0.87)
- Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks (ICLR 2024 · similarity 0.87)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.86)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
