Research map / Label-efficient and robust learning / Self-supervised and contrastive

LEAD: Exploring Logit Space Evolution for Model Selection

CVPR 2024 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervised neuroimaging representations · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LEAD, a transferability metric that models fine-tuning as a logit-space ODE solved in closed form via NTK eigenvalues with class-aware decomposition, to rank pre-trained vision models in one step.

Problem

Efficiently predicting the transferability of pre-trained vision models (model selection) without performing full fine-tuning, by modeling fine-tuning dynamics more faithfully than existing feature-space linear simulations.

Approach

Model the evolution of network output logits (rather than features) with an ODE derived from gradient descent and NTK, giving a closed-form interpolation between initial logits and one-hot labels; use per-class NTK eigenvalues as convergence rates, then score transferability by cross-entropy of predicted final logits.

Key results

• On the supervised model zoo of 12 CNNs, LEAD achieves average weighted Kendall's tau of 0.747, about 18% relatively better than the prior SOTA ETran.
• On the self-supervised model zoo (12 SSL models), LEAD achieves average tau of 0.746, about 15% relatively better than PED (0.648).
• In the low-data regime with N/K=10, LEAD reaches average tau 0.64 on supervised models vs 0.56 for ETran, using under 5% of the data.

Measured effects reported in the paper

Surprising findings

• Simulation-based methods (SFDA, PED) can fail to produce valid rankings with very few samples per class because all models perfectly separate the data, yielding identical scores.

Limitations

• Derivation relies on NTK infinite-width theory; finite-width approximation is via eigenvalues of initial NTK, which is empirical.
• Scope limited to image classification downstream tasks.
• Ground-truth rankings depend on a grid search over learning rates and weight decay.

Experimental setup

Datasets: ImageNet Caltech-101 Stanford Cars CIFAR10 CIFAR100 DTD Oxford 102 Flowers Food-101 Oxford-IIIT Pets SUN397 VOC2007

Benchmarks: Caltech-101 Stanford Cars CIFAR10 CIFAR100 DTD Oxford 102 Flowers Food-101 Oxford-IIIT Pets SUN397 VOC2007

Metrics: weighted Kendall's tau running time logit prediction bias (Euclidean distance)

Baselines: PARC LogME NLEEP PACTran SFDA GBC ETran PED

Builds on: Neural Tangent Kernel SFDA PED LogME PACTran

Closest papers by meaning

• Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods (ICML 2025 · similarity 0.86)
• PROFIT: A Specialized Optimizer for Deep Fine Tuning (NeurIPS 2025 · similarity 0.86)
• Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.86)
• Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.85)
• PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.85)
• Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.85)