Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Unsupervised Meta-Learning via In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAMeLU, an unsupervised meta-learning method that trains a transformer encoder for in-context prediction of query images, using augmentation-based support sets and mixup-style query generation, evaluated cross-domain without test-time fine-tuning.
Problem
Unsupervised meta-learning learns transferable representations from unlabeled data, but existing methods use simple augmentations that create a train/test task mismatch and typically evaluate only in-domain, requiring fine-tuning at test time.
Approach
Reframe meta-learning as non-causal sequence modeling in which a transformer encoder predicts query labels in-context from support embeddings plus learned class embeddings, with tasks built by augmenting support images and creating queries via a mixup-like blend of an augmented support image with a random training image, all labeled with the support pseudo-label.
Key results
- CAMeLU reaches 76.51% (5w1s) and 92.14% (5w5s) on mini-ImageNet, and 65.52% / 80.35% on CUB, outperforming the UML baselines in the table including cross-domain PsCo.
- CAMeLU outperforms its supervised counterpart CAML on CUB, Aircraft, and Meta-iNat (e.g., CUB 5w1s 65.52 vs 54.63), while CAML is higher on mini-ImageNet 5w1s (81.75 vs 76.51).
- With a small-scale mini-ImageNet training set, CAMeLU achieves 60.60% on CUB 5w1s, versus 42.47% for BECLR and 33.09% for PsCo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAMeLU instead of cross-domain PsCo | Unsupervised few-shot classification · miniImageNet, CIFAR-fs, CUB, Aircraft, Meta-iNat; 5-way 1-shot and 5-way 5-shot | accuracy | Helped |
| Train CAMeLU with ImageNet-964 + MSCOCO + Fungi instead of ImageNet-964 alone (ablation) | CAMeLU · Cross-domain few-shot classification | accuracy | No clear effect |
| Replace ResNet-50 (ImageNet-964) with ViT-B/16 pretrained with CLIP (ablation) | CAMeLU · CIFAR-fs, CUB, Aircraft, Meta-iNat | accuracy | Helped |
| Replace ResNet-50 (ImageNet-964) with ImageNet-1k SimCLR-pretrained ResNet-50 (ablation) | CAMeLU · Cross-domain few-shot classification | accuracy | Hurt |
| Replace ResNet-50 (ImageNet-964) with ImageNet-1k SwAV-pretrained ResNet-50 (ablation) | CAMeLU · Cross-domain few-shot classification | accuracy | Hurt |
| Replace ResNet-50 (ImageNet-964) with ResNet-50 pretrained on ImageNet-1k (ablation) | CAMeLU · miniImageNet | accuracy | Mixed |
| Generate query samples using the proposed image-mixing task strategy rather than augmentations alone (ablation) | CAMeLU · CUB, Aircraft, Meta-iNat | accuracy | Helped |
| Use the proposed task creation strategy instead of k-means clustering (ablation) | CAMeLU · Cross-domain classification | accuracy | Helped |
| Add the transformer in-context learner to the task creation mechanism (ablation) | CAMeLU · Few-shot classification, particularly Aircraft | accuracy | Helped |
Surprising findings
- CAMeLU outperforms its supervised counterpart CAML on several cross-domain datasets despite not using labels.
- Training on a multi-dataset mix improved CAML but not CAMeLU.
Limitations
- Baselines were trained with a smaller Conv5 or differing feature extractors, and SSL baselines use ImageNet-1k pretraining with data leakage on mini-ImageNet, complicating comparison.
- Results with ImageNet-1k-pretrained extractors on mini-ImageNet are affected by data leakage; CLIP ViT-B/16 training data cannot be verified.
- Relies on the assumption that augmentations preserve class membership, which cannot be verified on unlabeled data.
Experimental setup
Datasets: ImageNet-964 mini-ImageNet CIFAR-fs CUB Aircraft Meta-iNat MSCOCO Fungi
Benchmarks: mini-ImageNet CIFAR-fs CUB Aircraft Meta-iNat
Metrics: accuracy (5-way 1-shot) accuracy (5-way 5-shot)
Baselines: CACTUs-MAML CACTUs-ProtoNet UMTRA Meta-GMVAE PsCo BECLR CAML SimCLR SwAV
Builds on: CAML mixup UMTRA CACTUs GPICL
Closest papers by meaning
- Unsupervised Learning via Meta-Learning (ICLR 2019 · similarity 0.92)
- Meta-Learning without Memorization (ICLR 2020 · similarity 0.90)
- Context-Aware Meta-Learning (ICLR 2024 · similarity 0.89)
- An Information-Theoretic Analysis of In-Context Learning (ICML 2024 · similarity 0.87)
- In-context Learning in Presence of Spurious Correlations (NeurIPS 2026 · similarity 0.87)
- In-Context Learning with Representations: Contextual Generalization of Trained Transformers (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.