Research map / Architectures and efficiency / Pruning and efficient tuning
Effective pruning of web-scale datasets based on complexity of concept clusters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales SSP-Pruning to LAION and DataComp by deduplicating, CLIP-score filtering, and pruning with a concept-complexity-weighted density-based pruning (DBP), training CLIP ViT-B/32 on a fraction of the data for lower compute.
Problem
Pruning web-scale multimodal datasets to train CLIP-style models efficiently; existing best pruning (SSP-Pruning) was designed for ImageNet and does not account for concept complexity at web scale.
Approach
Density-Based Pruning allocates per-cluster sample counts via a softmax over a complexity score (product of inter-cluster and intra-cluster cosine distances), keeps the least prototypical samples within each cluster, and applies deduplication and CLIP-score filtering beforehand.
Key results
- On LAION, CLIP-B/32 trained on a 112M DBP subset reaches 65.44% ImageNet zero-shot vs 62.92% for OpenCLIP at 27.7% of the training cost (abstract reports 1.1 p.p. gain).
- On DataComp Medium, the DeDup 80% + CLIP-L/14 50% + DBP (19.2M) pipeline reaches 33.35% ImageNet zero-shot, beating T-MARS (33.00%) on ImageNet and achieving 37.26 VTAB.
- DBP outperforms SSP-Pruning on LAION at both 112M (49.8 vs 48.6) and 166M (51.6 vs 50.7) average zero-shot accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DBP instead of SSP-Pruning | CLIP ViT-B/32 trained on LAION-CAT-440M · Average zero-shot performance on 38 tasks, 112M | average zero-shot performance: 48.6 -> 49.8 | Helped |
| Use DBP instead of CLIP-score filtering | CLIP models trained on LAION-50M · ImageNet, 5 epochs | top-1 accuracy: 32.32 -> 39.04 | Helped |
| Use DBP with DeDup 80% and CLIP-L/14 Score 40% instead of the Image-based ∩ CLIP Score top 30% baseline | CLIP ViT-B/32 on DataComp Medium · Average, 38 tasks | average accuracy: 32.89 -> 35.35 | Helped |
| Deduplicate to 80% before DBP (ablation) | CLIP ViT-B/32 on DataComp Medium · ImageNet, DBP dataset size 48M | top-1 accuracy: 20.70 -> 21.34 | Helped |
| Deduplicate to 80% before DBP (ablation) | CLIP ViT-B/32 on DataComp Medium · Average, DBP dataset size 48M | average accuracy: 27.00 -> 27.71 | Helped |
| Deduplicate to 80% before CLIP-score filtering (ablation) | CLIP ViT-B/32 on DataComp Medium · ImageNet, 38M | top-1 accuracy: 27.30 -> 27.93 | Helped |
| Deduplicate to 80% before CLIP-score filtering (ablation) | CLIP ViT-B/32 on DataComp Medium · Average, 38M | average accuracy: 32.80 -> 33.03 | Helped |
| Increase training from 41.6% to 69% of the training cost (ablation) | CLIP ViT-B/32 trained on the 166M LAION subset · ImageNet dist. shifts | performance: +0.9p.p. | Helped |
| Increase training from 41.6% to 69% of the training cost (ablation) | CLIP ViT-B/32 trained on the 166M LAION subset · Retrieval | performance: +0.8p.p. | Helped |
| Use distilled DINOV2-L/14 image embeddings instead of the tested alternative encoders or embedding modalities (ablation) | DBP on LAION-50M · ImageNet | zero-shot accuracy | Helped |
| Set cluster balancing ratio to zero instead of using cluster balancing (ablation) | DBP on LAION-50M · ImageNet | zero-shot performance | Helped |
| Use 50% rather than 40% CLIP-L/14 Score before DBP (ablation) | CLIP ViT-B/32 on DataComp Medium · ImageNet dist. shifts | accuracy: 20.17 -> 17.73 | Mixed |
Surprising findings
- Cluster balancing is unnecessary for DBP: the best result was at a balancing ratio of zero.
- Deduplication is a necessary precursor for DBP and CLIP-score filtering; without it, pruning on DataComp performed worse.
Limitations
- Retrieval and ImageNet distribution shift performance lags at shorter training budgets, partially attributed to fewer iterations.
- Pruning hyperparameters (k=500, tau=0.1, l=20) were tuned on LAION-50M and transferred to other settings with limited re-tuning.
- T-MARS outperforms the method on ImageNet distribution shift tasks on DataComp.
Experimental setup
Datasets: LAION-CAT-440M LAION-2B LAION-50M DataComp Medium
Benchmarks: ImageNet ImageNet distribution shifts VTAB retrieval (Flickr, MSCOCO, WinoGAViL) DataComp 38-task suite
Metrics: zero-shot top-1 accuracy average zero-shot accuracy recall Jaccard score training cost %
Baselines: OpenCLIP LAION-440M training SSP-Pruning SemDeDup CLIP-score filtering T-MARS Image-based ∩ CLIP Score (L/14 top 30%)
Builds on: SSP-Pruning SemDeDup CLIP OpenCLIP DataComp
Closest papers by meaning
- Exploring Learning Complexity for Efficient Downstream Dataset Pruning (ICLR 2025 · similarity 0.89)
- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty (ICML 2025 · similarity 0.87)
- DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models (CVPR 2025 · similarity 0.87)
- No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance (NeurIPS 2024 · similarity 0.86)
- Sieve: Multimodal Dataset Pruning using Image Captioning Models (CVPR 2024 · similarity 0.86)
- Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.