Research map / Vision-language and multimodal / CLIP and zero-shot
Active Data Curation Effectively Distills Large-Scale Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACID, online batch selection using a larger pretrained reference model as implicit distillation for contrastive VLM pretraining, and ACED, which combines it with explicit softmax KD to yield FLOP-efficient CLIP-style encoders.
Problem
Compressing large contrastive vision-language models into smaller, inference-efficient two-tower models without losing zero-shot and retrieval performance, where prior work assumes large reference/teacher models cannot select data well for small students.
Approach
Active (online joint batch) data curation using a larger reference model's loss acts as an implicit form of distillation (ACID); it is complementary to explicit softmax KD, and combining them (ACED, ACIDistill) outperforms both.
Key results
- ACED-F0 averages 64.0 over 27 StableEval tasks versus 63.6 for MobileCLIP-S0 at 10.81% fewer inference GFLOPs.
- ACED-F1 reaches 69.7 average versus 67.9 for MobileCLIP-S1 with 7.14 vs 7.64 GFLOPs; ACED-F2 reaches 70.9 versus 69.8 for MobileCLIP-S2.
- In LiT-decoder evaluation, ACED (B/16) improves Flickr30k captioning CIDEr to 55.5 vs 53.4 for SigLIP-B/16 while using fewer image GFLOPs (23.19 vs 23.45).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACID active batch selection instead of Softmax-KD | ViT-S student · StableEval | average performance | Helped |
| Use H-ACID rather than I-ACID sampling (ablation) | ACID · StableEval | performance | Helped |
| Increase reference model capacity up to the optimal student-reference pairing (ablation) | ACID · StableEval | performance | Mixed |
| Use a larger reference model for ACID sampling (ablation) | ACID · StableEval | performance | Helped |
| Use a larger teacher for Softmax-KD (ablation) | Softmax-KD · StableEval | performance | Mixed |
| Change the reference/teacher pretraining dataset (ablation) | ACID and Softmax-KD · StableEval | performance | Helped |
| Replace Softmax-KD with Sigmoid-KD, feature-matching KD, or combined KD objectives (ablation) | ViT-S student · StableEval | performance | Helped |
| Change student size among Ti, S, and B (ablation) | ACID and Softmax-KD · StableEval | performance | Mixed |
| Combine ACID sampling with explicit Softmax-KD on the same sampled batch (ACIDistill) (ablation) | ACED · StableEval | performance | Helped |
| Use IID sampling for the distillation batch instead of ACID sampling (IIDistill) (ablation) | ACED · StableEval | performance | Hurt |
| Combine active curation with explicit distillation (ablation) | ACED · StableEval | performance | Helped |
| Use ACID instead of Softmax-KD | small multimodal student · Cars and DTD | performance | Hurt |
Surprising findings
- Larger reference models can effectively curate data for smaller students, contrary to prior online batch selection work that used same-size or smaller references.
- Softmax-KD only benefits from much larger teachers, while ACID outperforms the IID baseline even with a tiny reference model.
Limitations
- Training cost of large reference/teacher models and super-batch scoring is substantial and not fully accounted for in the main text.
- ACID underperforms Softmax-KD on some fine-grained evaluations (e.g., Cars, DTD), motivating the combined method.
- Evaluated mainly on contrastive two-tower VLMs with ViT-S/BERT-small students and zero-shot classification and retrieval.
Experimental setup
Datasets: DataComp-1B WebLI-curated++ WebLI-1B LAION-400M
Benchmarks: ImageNet ImageNet distribution shifts COCO retrieval Flickr30k retrieval StableEval 27 zero-shot and retrieval evals Flickr30k captioning VQAv2 GQA
Metrics: zero-shot top-1 accuracy recall@k average performance over 27 evals CIDEr VQA accuracy inference GFLOPs
Baselines: Softmax-KD Sigmoid-KD Feature-Matching KD CLIP-KD MobileCLIP TinyCLIP CLIP-CID DatologyAI-CLIP SigLIP SiLC
Builds on: RHO-Loss Evans et al. joint example selection CLIP SigLIP knowledge distillation (Hinton)
Closest papers by meaning
- Data curation via joint example selection further accelerates multimodal learning (NeurIPS 2024 · similarity 0.89)
- Efficient Multimodal Dataset Distillation via Generative Models (NeurIPS 2025 · similarity 0.87)
- CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder (NeurIPS 2025 · similarity 0.86)
- Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation (NeurIPS 2026 · similarity 0.86)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.86)
- MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.