Research map / Vision-language and multimodal / CLIP and zero-shot
Data Adaptive Traceback for Vision-Language Foundation Models in Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Data Adaptive Traceback (DAT), which zero-shot samples task-relevant pre-training image-text pairs and reuses them via pseudo-label semi-supervised learning plus a vision-language contrastive loss to adapt CLIP to downstream image classification.
Problem
Vision-language foundation models pre-trained on web image-text pairs contain many weakly paired samples, so they fail to fully capture knowledge relevant to downstream tasks, and existing adaptation methods ignore this missing knowledge.
Approach
Reuse the pre-training images at adaptation time: a two-step CLIP zero-shot sampler (class-level, then per-downstream-image) selects task-related pre-training data, FixMatch-style pseudo-labeling trains on it, and a UniCL-style image-text contrastive loss over a unified image-text-label space mitigates confirmation bias.
Key results
- With an RN50 OpenCLIP encoder pre-trained on YFCC15M, DAT reaches 82.48% average accuracy over eight benchmarks versus 80.97% for fine-tuning.
- With ViT-B-32 and a 30M LAION-400M subset, DAT-finetuning averages 85.75% versus 82.18% for fine-tuning, a gain of about 3.57 points.
- Ablation on DTD and Oxford Pets shows the data sampling module (DAT sampling vs random sampling) yields 73.88 vs 73.14 on DTD and 87.98 vs 86.32 on Oxford Pets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reuse pre-training data with DAT during fine-tuning | OpenCLIP ResNet-50, YFCC15M · CIFAR10 | accuracy: 95.52 -> 96.21 | Helped |
| reuse pre-training data with DAT during fine-tuning | OpenCLIP ResNet-50, YFCC15M · CIFAR100 | accuracy: 79.50 -> 81.13 | Helped |
| reuse pre-training data with DAT during fine-tuning | OpenCLIP ResNet-50, YFCC15M · FGVC Aircraft | accuracy: 78.37 -> 78.85 | Helped |
| reuse pre-training data with DAT during fine-tuning | OpenCLIP ResNet-50, YFCC15M · Oxford Pets | accuracy: 85.53 -> 87.98 | Helped |
| reuse pre-training data with DAT during fine-tuning | OpenCLIP ResNet-50, YFCC15M · Stanford Cars | accuracy: 82.08 -> 83.75 | Helped |
| reuse pre-training data with DAT during fine-tuning | OpenCLIP ResNet-50, YFCC15M · DTD | accuracy: 70.59 -> 73.88 | Helped |
| reuse pre-training data with DAT during fine-tuning | OpenCLIP ResNet-50, YFCC15M · Food101 | accuracy: 86.90 -> 87.90 | Helped |
| reuse pre-training data with DAT during fine-tuning | OpenCLIP ResNet-50, YFCC15M · SUN397 | accuracy: 69.26 -> 70.17 | Helped |
| use DAT with full fine-tuning | ViT-B-32, LAION-400M · CIFAR100 | accuracy: 86.08 -> 88.08 | Helped |
| use DAT with full fine-tuning | ViT-B-32, LAION-400M · DTD | accuracy: 72.34 -> 79.01 | Helped |
| replace random sampling with the proposed data sampling module (ablation) | DAT · Oxford Pets | accuracy: 86.32 -> 87.98 | Helped |
| replace random sampling with the proposed data sampling module (ablation) | DAT · DTD | accuracy: 73.14 -> 73.88 | Helped |
Surprising findings
- Combining DAT with bias-tuning or adapter-tuning gives little gain over those methods alone, which the authors attribute to frozen parameters limiting learning of new knowledge.
Limitations
- Pseudo-labels from thresholding are of low quality, causing fluctuating results.
- Pre-training text contains substantial noise, which makes training difficult and can cause convergence issues.
- Results on the larger LAION setting use only a 30M-pair subset due to compute limits.
Experimental setup
Datasets: YFCC15M LAION-400M WebImageText-400M CIFAR10 CIFAR100 FGVC Aircraft Oxford Pets Stanford Cars DTD Food101 SUN397
Benchmarks: CIFAR10 CIFAR100 FGVC Aircraft Oxford Pets Stanford Cars DTD Food101 SUN397
Metrics: top-1 accuracy
Baselines: Fine-tuning FixMatch CCSSL linear-probing bias-tuning adapter-tuning
Builds on: CLIP FixMatch UniCL CCSSL OpenCLIP
Closest papers by meaning
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.88)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.88)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.88)
- Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model (ICLR 2025 · similarity 0.87)
- Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models (CVPR 2024 · similarity 0.87)
- Bayesian Test-Time Adaptation for Vision-Language Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.