Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Let Go of Your Labels with Unsupervised Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TURTLE, a fully unsupervised transfer method that searches for the labeling of a dataset maximizing linear-classifier margins across one or two frozen foundation-model representation spaces (CLIP plus DINOv2).
Problem
Unsupervised transfer from foundation models: inferring the underlying human labeling of a downstream dataset given only the number of classes, without class descriptions, labels, or task-specific representation learning.
Approach
Shows the generalization-based objective (HUME-style bilevel) implicitly maximizes the max-margin of linear classifiers over labelings; TURTLE directly maximizes margins in frozen foundation-model spaces (an ensemble of task encoders over one or more representations) with entropy regularization, using first-order hypergradient approximation (no unrolling) for efficiency.
Key results
- TURTLE 2-spaces outperforms CLIP zero-shot by 9%, 7%, and 4% absolute with ViT-B/32, ViT-B/16, and ViT-L/14 backbones respectively, without any supervision.
- TURTLE reaches 72.9% unsupervised accuracy on ImageNet, outperforming the previous state of the art by 5.5%.
- On the ResNet-50 prompt-tuning comparison, TURTLE averages 71.2% vs. 63.5% for the best unsupervised prompt tuning baseline (GDA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TURTLE 2-spaces instead of the strongest unsupervised prompt-tuning baseline | Unsupervised transfer with CLIP RN50 features · Average across 11 datasets | accuracy: 8% absolute improvement | Helped |
| Use TURTLE 2-spaces instead of unsupervised prompt-tuning baselines | Unsupervised transfer with CLIP RN50 features · Flowers102 | accuracy: 27% | Helped |
| Use TURTLE 2-spaces instead of unsupervised prompt-tuning baselines | Unsupervised transfer with CLIP RN50 features · EuroSAT | accuracy: 41% | Helped |
| Use TURTLE 2-spaces instead of CLIP zero-shot transfer | ViT-B/32 · 26-dataset benchmark | accuracy: 9% absolute improvement | Helped |
| Use TURTLE 2-spaces instead of CLIP zero-shot transfer | ViT-B/16 · 26-dataset benchmark | accuracy: 7% absolute improvement | Helped |
| Use TURTLE 2-spaces instead of CLIP zero-shot transfer | ViT-L/14 · 26-dataset benchmark | accuracy: 4% absolute improvement | Helped |
| Use TURTLE 2-spaces instead of CLIP zero-shot transfer | TURTLE with CLIP and DINOv2 representations · EuroSAT | accuracy: 35% | Helped |
| Use TURTLE 2-spaces instead of CLIP zero-shot transfer | TURTLE with CLIP and DINOv2 representations · MNIST | accuracy: 21% | Helped |
| Use TURTLE 2-spaces instead of CLIP zero-shot transfer | TURTLE with CLIP and DINOv2 representations · Flowers102 | accuracy: 20% | Helped |
| Add a stronger foundation-model representation space (ablation) | TURTLE · ImageNet-1000 | accuracy | Helped |
| Use the efficient first-order hypergradient approximation | TURTLE · ImageNet | training time: 10× speedup | Helped |
| Use TURTLE instead of HUME | Unsupervised transfer · MNIST | accuracy: 23% absolute improvement | Helped |
Surprising findings
- Fully unsupervised TURTLE with a single CLIP space roughly matches CLIP zero-shot (which uses class names) across ViT backbones.
- TURTLE is about 10x faster than HUME, finishing ImageNet in under five minutes.
Limitations
- Performance depends on representation quality; gains are tied to strong foundation models (correlation with linear probe).
- Requires the number of classes to be known a priori.
- Model selection relies on cross-validation over 50 hyperparameter configurations per dataset, with correlation to accuracy of about 0.61 on average.
Experimental setup
Datasets: CIFAR10 CIFAR100 STL10 ImageNet Caltech101 Food101 Flowers102 Birdsnap StanfordCars FGVC Aircraft OxfordPets MNIST
Benchmarks: 26 vision datasets (ImageNet, CIFAR10, MNIST, EuroSAT, Flowers102, etc.)
Metrics: clustering accuracy (Hungarian-matched)
Baselines: HUME K-Means CLIP zero-shot UPL POUF GDA linear probe
Builds on: HUME generalization-based objective implicit bias of gradient descent on logistic regression CLIP DINOv2
Closest papers by meaning
- ZeroDiff: Solidified Visual-semantic Correlation in Zero-Shot Learning (ICLR 2025 · similarity 0.87)
- OTTER: Effortless Label Distribution Adaptation of Zero-shot Models (NeurIPS 2024 · similarity 0.87)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.87)
- Zero-Shot Robustification of Zero-Shot Models (ICLR 2024 · similarity 0.87)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.86)
- Label Propagation for Zero-shot Classification with Vision-Language Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.