Research map / Label-efficient and robust learning / Recognition and long-tail
When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an online training framework that unifies dynamic data selection with TrivialAugment, selecting samples by a joint density and CLIP-based semantic consistency distribution to cut training cost while keeping accuracy.
Problem
Dynamic data selection speeds up training but reduces data diversity and hurts generalization, while data augmentation is usually not optimized jointly with selection, and selected low-density samples may be noisy or ambiguous.
Approach
Score each sample by the product of its normalized local density (mean HNSW neighbor distance, computed online) and a precomputed CLIP image-label cosine consistency (via lightweight adapters), select high-scoring samples for training, and apply a single lightweight TrivialAugment transformation to them.
Key results
- On ImageNet-1k with ResNet-50, the method reaches 76.9% at a 60% selection ratio versus 76.4% for full data, with 88.2 vs 140.0 total GPU hours.
- On Tiny-ImageNet with noisy data (20% noise), accuracy reaches 35.9% at 20% selection versus 34.9% for InfoBatch and 33.8% for Random*.
- On CIFAR-10 with ResNet-18, 94.9% at 30% selection and 95.5% at 50%, compared with full-data 95.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dynamic selection and augmentation instead of InfoBatch | Ours · Tiny-ImageNet | accuracy (%): 43.2 -> 44.9 | Helped |
| Use dynamic selection and augmentation instead of InfoBatch | Ours · CIFAR-100 | accuracy (%): 76.5 -> 77.6 | Helped |
| Use dynamic selection and augmentation instead of InfoBatch | Ours · ImageNet-1k, 60% selection | accuracy (%): 76.5 -> 76.9 | Helped |
| Use dynamic selection and augmentation instead of InfoBatch | Ours · Tiny-ImageNet noisy, 20% noise, 20% selection | accuracy (%): 34.9 -> 35.9 | Helped |
| Use dynamic selection and augmentation instead of InfoBatch | Ours · Tiny-ImageNet corrupted, 20% selection | accuracy (%): 35.1 -> 39.1 | Helped |
| Add semantic consistency to density-based selection (ablation) | Ours · Tiny-ImageNet, 30% selection | accuracy (%): 39.0 -> 42.5 | Helped |
| Add the augmenter to density-plus-consistency selection (ablation) | Ours · Tiny-ImageNet, 30% selection | accuracy (%): 42.5 -> 43.5 | Helped |
| Add density to consistency-plus-augmentation selection (ablation) | Ours · Tiny-ImageNet, 30% selection | accuracy (%): 41.1 -> 43.5 | Helped |
| Add consistency to density-plus-augmentation selection (ablation) | Ours · Tiny-ImageNet, 30% selection | accuracy (%): 41.5 -> 43.5 | Helped |
| Add the augmenter to density-plus-consistency selection (ablation) | Ours · Tiny-ImageNet, 50% selection | accuracy (%): 46.3 -> 47.5 | Helped |
| Add the augmenter to density-plus-consistency selection (ablation) | Ours · Tiny-ImageNet, 70% selection | accuracy (%): 49.3 -> 50.2 | Helped |
Surprising findings
- Low-density samples alone degrade performance, so adding CLIP semantic consistency is needed to filter ambiguous or noisy samples before augmentation.
Limitations
- Relies on a pretrained CLIP model and image-label consistency, which may be limited for domains outside CLIP's coverage.
- Reported ImageNet-1k cost comparisons mix 8-A100 and 4-A100 setups and some baseline results are taken from prior work.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1k ImageNet-A ImageNet-O ImageNet-R ImageNet-Hard
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1k ImageNet-A ImageNet-O ImageNet-R ImageNet-Hard
Metrics: accuracy AUPR wall-clock time GPU hours
Baselines: Random Herding EL2N GraNd Glister Forgetting Moderate-DS Self-sup. prototypes MoSo DP UCB epsilon-Greedy
Builds on: InfoBatch TrivialAugment CLIP HNSW Moderate-DS
Closest papers by meaning
- A CLIP-Powered Framework for Robust and Generalizable Data Selection (ICLR 2025 · similarity 0.87)
- Data curation via joint example selection further accelerates multimodal learning (NeurIPS 2024 · similarity 0.87)
- SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification (NeurIPS 2024 · similarity 0.86)
- Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws (ICLR 2025 · similarity 0.86)
- DsDm: Model-Aware Dataset Selection with Datamodels (ICML 2024 · similarity 0.85)
- Hypernetworks for Dynamic Feature Selection (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.