Research map / Learning theory and algorithms / Prediction and decision losses
Diversified Batch Selection for Training Acceleration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DivBS, a reference-model-free online batch selection method that picks a diverse, representative subset by maximizing group-wise orthogonalized representativeness, with a greedy approximation and a cheaper selection variant.
Problem
Online batch selection for training acceleration without reference models tends to be sample-wise, producing redundant, low-diversity subsets that hurt performance, especially under small budgets.
Approach
Treat the selected subset as a whole: score a subset by its orthogonalized representativeness of the batch (projection of the batch gradient-feature sum onto the subset's span), which removes inter-sample redundancy, and optimize it greedily with a 1-1/e approximation guarantee plus a cheaper Sum-based selection rule.
Key results
- On CIFAR-10 with 10% budget, DivBS reaches 94.65% accuracy vs 92.06% for uniform sampling (full data 95.50%).
- On CIFAR-100 with 30% budget, DivBS needs 52 epochs to reach 62% accuracy vs 113 for uniform sampling.
- DivBS keeps selection overhead under 5% of total time versus uniform sampling and improves wall-clock speed-up trade-off.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select batches with DivBS rather than CCS-BS | Online batch selection with ResNet-18 · CIFAR-10, 10% budget | accuracy: 92.61% -> 94.65% | Helped |
| Select batches with DivBS rather than CCS-BS | Online batch selection with ResNet-18 · CIFAR-100, 10% budget | accuracy: 71.11% -> 73.11% | Helped |
| Select batches with DivBS rather than Moderate-BS | Online batch selection with ResNet-18 · Tiny ImageNet, 10% budget | accuracy: 48.92% -> 50.84% | Helped |
| Select batches with DivBS rather than Uniform | Online batch selection with ResNet-18 · CIFAR-100-LT, imbalance ratio 100, 10% budget | final accuracy: 26.97% -> 31.74% | Helped |
| Select batches with DivBS rather than Uniform | Online batch selection for semantic segmentation · PASCAL VOC 2012 trainaug, 10% budget | final mIoU: 63.72% -> 65.45% | Helped |
| Select batches with DivBS rather than Uniform | Cross-modal retrieval training · Wikipedia, 30% budget | average MAP: 0.491 -> 0.495 | Helped |
| Select batches with DivBS rather than Uniform | GPT-2 Medium fine-tuning with LoRA · E2E NLG Challenge, 20% budget | BLEU: 66.14% -> 66.87% | Helped |
| Use the efficient selection realization that subtracts orthogonal components from Sum, instead of orthogonalizing all candidate features (ablation) | DivBS · Online batch selection | selection time | Helped |
| Use the efficient DivBS selection instead of Uniform sampling | CIFAR-10 training with ResNet-18 · CIFAR-10, 10% budget | wall-clock time per epoch: 13.2s -> 13.9s | Hurt |
| Use DivBS rather than Uniform sampling | CIFAR-100 training with AdamW · CIFAR-100, 10% budget | accuracy: 66.72% -> 69.07% | Helped |
Surprising findings
- Most reference-model-free baselines (Train Loss, Grad Norm, SVP) fall well below uniform sampling on CIFAR-100 and Tiny ImageNet.
- On the cross-modal retrieval task, DivBS surpasses full-data training in text-to-image and average MAP at 20% and 30% budgets.
Limitations
- Online batch selection needs an extra forward pass per batch to compute selection features, adding overhead.
- Evaluated mainly on small-scale vision benchmarks and a single LM fine-tuning setup; not on large-scale pre-training.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet CIFAR-100-LT PASCAL VOC 2012 trainaug Wikipedia E2E NLG Challenge
Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet CIFAR-100-LT PASCAL VOC 2012 trainaug Wikipedia E2E NLG Challenge
Metrics: accuracy epochs to target accuracy mIoU MAP BLEU NIST METEOR ROUGE-L CIDEr wall-clock time
Baselines: Uniform Train Loss Grad Norm Grad Norm IS SVP Moderate-BS CCS-BS InfoBatch K-means++ initialization RHO-LOSS Bayesian
Builds on: online batch selection submodular maximization greedy Selection-via-Proxy RHO-LOSS
Closest papers by meaning
- GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration (NeurIPS 2024 · similarity 0.87)
- Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyond (ICML 2024 · similarity 0.86)
- DsDm: Model-Aware Dataset Selection with Datamodels (ICML 2024 · similarity 0.86)
- PEAKS: Selecting Key Training Examples Incrementally via Prediction Error Anchored by Kernel Similarity (ICML 2025 · similarity 0.86)
- Most Influential Subset Selection: Challenges, Promises, and Beyond (NeurIPS 2024 · similarity 0.85)
- BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Ranges (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.