AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Towards Inadequately Pre-trained Models in Transfer Learning

Andong Deng, Xingjian Li, Di Hu, Tianyang Wang, Haoyi Xiong, Cheng-Zhong Xu

ICCV 2023 · Architectures and efficiency › Pruning and efficient tuning · approach: Parameter-efficient transfer analysis · 18 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
stop pre-training early (middle epoch) when using the model as a frozen feature extractorResNet-18 CIFAR10 to MNIST · MNIST (1000-sample toy)FE accuracy: 88.47 -> 96.47Improved
pre-train fully rather than stopping early when fine-tuning the whole modelResNet-18 CIFAR10 to MNIST · MNIST (full)FT accuracy: 99.30 -> 99.46Improved
longer pre-training on the main spectral component Fm for FE (ablation)ResNet-18 CIFAR10 to MNIST · MNIST FEFm accuracy: 88.24 -> 58.74Hurt
longer pre-training on residual component Fr for FE (ablation)ResNet-18 CIFAR10 to MNIST · MNIST FEFr accuracy: 55.45 -> 71.28Improved
use checkpoint selected at 70 epochs instead of fully trained 90 epochs as image feature backbonePVSE image-text retrieval · MSCOCOrsum: 132.67 -> 138.66Improved
use 70-epoch checkpoint for weight-imprinting few-shot classificationResNet50 on CUB-200-2011 · CUB-200-2011 N=1accuracy: 52.11 -> 52.92Improved
use transferability scores (LogME, LEEP, NCE) to select the best pre-training checkpointcheckpoint selection · FGVC Aircraft, CIFAR10, Flowers102correlation with FT performanceMixed
use self-supervised MoCo-v2 pre-training instead of supervisedfeature extractor transfer · selected datasetsFE accuracy vs epochImproved
train Swin-T longer for feature extractionSwin-T · CIFAR100, DTD, Flower102, Aircraft, IndoorFE accuracy: 75.72 (epoch 250) vs 75.58 (epoch 300) on CIFAR100No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy