Research map / Large language models / Language, safety and interpretability
Data Pruning by Information Maximization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfoMax, a coreset selection method that formulates data pruning as a discrete quadratic program trading off per-sample importance against pairwise redundancy, solved via a scalable gradient-style softmax iteration with sparsification and dataset partitioning.
Problem
Data pruning / coreset selection must pick a compact subset that stays informative while avoiding redundancy; score-based methods ignore overlap and greedy hybrid methods like D^2-Pruning get stuck in suboptimal solutions, and existing methods struggle to scale to millions of samples.
Approach
Cast coreset selection as a discrete quadratic program maximizing the sum of per-sample information scores minus pairwise similarity penalties among selected samples (derived from a graph-cut conditional gain view of set information), relax it to a continuous problem, and solve with a proximal-style softmax update, plus k-NN sparsification of the similarity matrix and random dataset partitioning f
Key results
- On CIFAR-100 at 10% selection, InfoMax reaches 58.7 Acc@1 vs 56.9 for D^2-Pruning (supervised setting).
- On CC12M at 10% selection, InfoMax outperforms D^2-Pruning by 1.5%, 1.3%, 2.7%, and 1.0% across the four downstream CLIP tasks.
- On instruction tuning with LLaMA2-7B, InfoMax reaches 47.2 on BBH at 5% selection vs 41.7 for D^2-Pruning; processes 12M samples in about 37 minutes on two 8-GPU nodes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InfoMax instead of D²-Pruning for 10% selection | CIFAR-10 coreset selection · CIFAR-10, 10% selection | accuracy: 87.1 -> 89.1 | Helped |
| Use InfoMax instead of CCS for 10% selection | CIFAR-100 coreset selection · CIFAR-100, 10% selection | accuracy: 57.3 -> 58.7 | Helped |
| Use InfoMax instead of CCS for 10% selection | ImageNet-1K coreset selection · ImageNet-1K, 10% selection | accuracy: 57.3 -> 59.0 | Helped |
| Use InfoMax instead of D²-Pruning | LLaMA2-7B instruction-tuning data selection · BBH, 5% selection | accuracy: 41.7 -> 47.2 | Helped |
| Use InfoMax instead of the second-ranked method | LLaMA2-7B instruction-tuning data selection · MMLU, 5% selection | accuracy: 51.8 vs. 50.2 | Helped |
| Reduce the size of each dataset partition (ablation) | InfoMax · CC3M and CC12M | accuracy | Hurt |
| Vary sparse rate k (ablation) | InfoMax · CC12M, zero-shot ImageNet-1K classification | accuracy | No clear effect |
| Vary pairwise weight α (ablation) | InfoMax · CC12M, zero-shot ImageNet-1K classification | accuracy | No clear effect |
| Increase solver iteration count T (ablation) | InfoMax · CC12M, zero-shot ImageNet-1K classification | accuracy | Helped |
Surprising findings
- The unsupervised variant (SSP scores with DINO features) matches or beats several supervised baselines, e.g., unsupervised InfoMax on ImageNet-1K at 10% outperforms supervised D^2-Pruning by 1.2%.
Limitations
- Largest experiments reach 12M samples; scaling to billion-scale datasets is untested.
- Scalability depends on partition size; performance declines when partitions are smaller than about 1M samples.
- Efficiency is slower than pure score-based methods due to similarity matrix construction and feature extraction.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K CC12M CC3M FLAN-V2 CoT Dolly Open-Assistant-1 Flickr30K
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K CC12M zero-shot ImageNet Flickr30K retrieval MMLU TydiQA BBH
Metrics: Acc@1 zero-shot accuracy linear probe accuracy image-to-text retrieval text-to-image retrieval accuracy
Baselines: Random Entropy Forgetting EL2N Moderate CCS D^2-Pruning K-center SSP CLIP score Clustering + CLIP LESS
Builds on: D^2-Pruning CCS SSP LESS submodular information measures
Closest papers by meaning
- $\mathbb{D}^2$ Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning (ICLR 2024 · similarity 0.92)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.87)
- Exploring Learning Complexity for Efficient Downstream Dataset Pruning (ICLR 2025 · similarity 0.87)
- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty (ICML 2025 · similarity 0.86)
- GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning (NeurIPS 2024 · similarity 0.86)
- InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.