Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Collaborative Consortium of Foundation Models for Open-World Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CO3, a training-light pipeline that combines frozen CLIP, DINO, GPT-3 and DALL-E to correct noisy support labels, augment data, and classify via a text-guided fusion adapter for open-world few-shot learning.
Problem
Open-world few-shot learning, where very few support samples carry unreliable labels with noise from seen or unseen classes.
Approach
Instead of training a feature extractor on base data, it orchestrates frozen foundation models (CLIP, DINO, GPT-3, DALL-E) for label correction and data augmentation, and trains only a TeFu-Adapter whose logits are regularized by CLIP text-image semantic similarity to resist label noise.
Key results
- On ImageNet 1-shot with 0.3 noisy label proportion, CO3 reaches 62.86% accuracy, versus 59.97% for CaFo... (CaFo at 0.3 is 59.99%) and 59.97% for Tip-Adapter-F at 0.3 is 60.35%.
- At 100% noisy labels on ImageNet 1-shot, CO3 maintains 62.58% while CoOp drops to 4.57% and CLIP-Adapter to 18.85%.
- Ablation on ImageNet 1-shot (0.3 noise): TeFu-Adapter achieves 62.86% versus 59.45% for Tip-Adapter structure at 0.3 noise ratio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CO3 instead of CaFo | OFSL methods · ImageNet, 1-shot, noisy label proportion 0.3 | accuracy: 59.99 -> 62.86 | Helped |
| use CO3 instead of the strongest listed baseline | OFSL methods · ImageNet, 1-shot, noisy label proportion 1.0 | accuracy: 59.03 -> 62.58 | Helped |
| add the LC-Block as an auxiliary module and use threshold-based label correction (ablation) | CO3 · ImageNet, 1-shot | accuracy: at least 1.8% | Helped |
| add DALL-E data augmentation with corrected category names (ablation) | CO3 · ImageNet, 1-shot | accuracy: approximately 1.2% | Helped |
| fuse CLIP and DINO features instead of using CLIP features alone (ablation) | CO3 feature extraction · ImageNet, 1-shot | accuracy: 3.5% | Helped |
| use DINO features alone instead of integrating CLIP and DINO features (ablation) | CO3 feature extraction · ImageNet, 1-shot | accuracy: around 30% | Hurt |
| replace TeFu-Adapter with no adapter (ablation) | CO3 · ImageNet, 1-shot, noisy label proportion 0.3 | accuracy: 15.17 -> 62.86 | Hurt |
| replace TeFu-Adapter with Tip-Adapter (ablation) | CO3 · ImageNet, 1-shot, noisy label proportion 0.3 | accuracy: 59.45 -> 62.86 | Hurt |
| replace TeFu-Adapter with CLIP-Adapter (ablation) | CO3 · ImageNet, 1-shot, noisy label proportion 0.3 | accuracy: 56.33 -> 62.86 | Hurt |
| replace TeFu-Adapter with no adapter (ablation) | CO3 · ImageNet, 1-shot, noisy label proportion 0.9 | accuracy: 15.92 -> 62.50 | Hurt |
| replace TeFu-Adapter with Tip-Adapter (ablation) | CO3 · ImageNet, 1-shot, noisy label proportion 0.9 | accuracy: 52.29 -> 62.50 | Hurt |
| replace TeFu-Adapter with CLIP-Adapter (ablation) | CO3 · ImageNet, 1-shot, noisy label proportion 0.9 | accuracy: 30.08 -> 62.50 | Hurt |
Surprising findings
- Using the LC-Block alone for classification yields poor results, and relying solely on DINO features drops accuracy to about 30%.
Limitations
- Relies on several proprietary or large generative models (GPT-3, DALL-E) whose cost and access are not analyzed.
- Results reported mainly on five datasets; the paper says other datasets are covered but only four are shown.
- Performance inconsistencies noted in text, such as the 0.3-noise claim, make some comparisons hard to verify.
Experimental setup
Datasets: ImageNet OxfordPets Caltech101 Food101 Sun397
Benchmarks: ImageNet OxfordPets Caltech101 Food101 Sun397
Metrics: accuracy
Baselines: CoOp Tip-Adapter-F CLIP-Adapter CALIP-FS CaFo APE-T
Builds on: CaFo Tip-Adapter CLIP DINO
Closest papers by meaning
- DeIL: Direct-and-Inverse CLIP for Open-World Few-Shot Learning (CVPR 2024 · similarity 0.87)
- Scaling Few-Shot Learning for the Open World (AAAI 2024 · similarity 0.84)
- Pushing the Limit of Fine-Tuning for Few-Shot Learning: Where Feature Reusing Meets Cross-Scale Attention (AAAI 2024 · similarity 0.83)
- LR0.FM: Low-Res Benchmark and Improving robustness for Zero-Shot Classification in Foundation Models (ICLR 2025 · similarity 0.83)
- Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning (CVPR 2025 · similarity 0.83)
- S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.