Research map / Architectures and efficiency / RNNs and normalization
Swift Sampler: Efficient Learning of Sampler by 10 Parameters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Swift Sampler (SS), a Bayesian-optimization-based search that parameterizes a data sampler with 10 hyperparameters (piecewise-linear map over loss/entropy features with a cumulative-gradient transform) and approximates local minima by fine-tuning a shared pre-trained checkpoint for 5 epochs.
Problem
Automatically learning data sampling probabilities is costly: sample-based search over per-instance probabilities is high-dimensional, the objective is sharp, and evaluating each sampler requires training from scratch, which is infeasible on large datasets like ImageNet.
Approach
Map a sampler to a low-dimensional family (10 hyperparameters) of functions of fixed, pre-computed features (loss and renormalized entropy) via a cumulative-gradient transform that smooths the objective, use GP-based Bayesian optimization with UCB, and evaluate candidates by briefly fine-tuning a shared pre-trained model rather than training from scratch.
Key results
- On ImageNet, SS improves Top-1 accuracy by 0.8% to 1.5% across eight architectures (e.g., ResNet-50 76.3 to 77.7).
- On CIFAR10 with 40% noise, SS reaches 88.0% vs 64.9% baseline and 86.9% for the best prior method (LR).
- Approximated-minima ranking gives average Spearman SR 0.72 and top-1 sampler ground-truth rank TR averaging 1.6 on CIFAR10 40% noise.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SS sampling instead of LR | ResNet-18 · CIFAR10, noise 0 | Top-1 accuracy: 91.0 -> 93.8 | Helped |
| Use SS sampling instead of LR | ResNet-18 · CIFAR100, noise 0 | Top-1 accuracy: 74.2 -> 75.2 | Helped |
| Use SS sampling instead of LR | ResNet-18 · CIFAR10, noise 0.1 | Top-1 accuracy: 91.0 -> 91.7 | Helped |
| Use SS sampling instead of LR | ResNet-18 · CIFAR100, noise 0.1 | Top-1 accuracy: 70.1 -> 71.2 | Helped |
| Use SS sampling instead of LR | ResNet-18 · CIFAR10, noise 0.2 | Top-1 accuracy: 89.2 -> 90.4 | Helped |
| Use SS sampling instead of LR | ResNet-18 · CIFAR100, noise 0.2 | Top-1 accuracy: 68.3 -> 69.2 | Helped |
| Use SS sampling instead of LR | ResNet-18 · CIFAR10, noise 0.3 | Top-1 accuracy: 88.5 -> 89.7 | Helped |
| Use SS sampling instead of LR | ResNet-18 · CIFAR100, noise 0.3 | Top-1 accuracy: 63.1 -> 64.4 | Helped |
| Use SS sampling instead of LR | ResNet-18 · CIFAR10, noise 0.4 | Top-1 accuracy: 86.9 -> 88.0 | Helped |
| Use SS sampling instead of LR | ResNet-18 · CIFAR100, noise 0.4 | Top-1 accuracy: 61.3 -> 62.3 | Helped |
| Use SS sampling instead of uniform sampling | ImageNet models · ImageNet ILSVRC12, Top-1 accuracy across listed architectures | Top-1 accuracy: 70.4 -> 71.9; 70.2 -> 71.6; 76.3 -> 77.7; 78.0 -> 79.3; 78.6 -> 79.8; 78.9 -> 80.0; 81.2 -> 82.1; 83.5 -> 84.3 | Helped |
| Use SS sampling instead of uniform sampling | ResNet-50 and ResNet-101 · YTF (trained on MS1M) | verification performance: 97.41 -> 97.50; 97.54 -> 97.74 | Helped |
Surprising findings
- Spearman rank correlation between approximated and from-scratch sampler rankings averages only 0.72 with a minimum of 0.43, while the top-1 sampler ground-truth rank averages 1.6.
- Samplers searched on small models (ResNet-18, ResNet-50) perform comparably to self-searched samplers when transferred to SE-ResNext-50 and SE-ResNext-101.
Limitations
- Gains are modest on Top-5 accuracy because only Top-1 is used as the search objective.
- Local-minimum approximation relies on fine-tuning a shared checkpoint from uniform sampling, and rank correlation with from-scratch training is only moderate on average.
- Appendix results (LAION-5B, Mini-ImageNet, Wikitext-2) are reported with few details and no variance, so their reliability is unclear.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet ILSVRC12 MS1M Wikitext-2 Mini-ImageNet LAION-5B
Benchmarks: CIFAR10 CIFAR100 ImageNet ILSVRC12 YTF
Metrics: Top-1 accuracy Top-5 accuracy Spearman rank correlation top-1 sampler ground-truth rank verification accuracy
Baselines: REED MentorNet Learning to reweight Random search Reinforcement learning sampler search Uniform sampling
Builds on: Bayesian Optimization Learning to reweight MentorNet REED
Closest papers by meaning
- Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning (ICLR 2024 · similarity 0.85)
- A CLIP-Powered Framework for Robust and Generalizable Data Selection (ICLR 2025 · similarity 0.85)
- Enhancing Semi-Supervised Learning via Representative and Diverse Sample Selection (NeurIPS 2024 · similarity 0.84)
- Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes (NeurIPS 2025 · similarity 0.83)
- Improving the Training of Data-Efficient GANs via Quality Aware Dynamic Discriminator Rejection Sampling (CVPR 2025 · similarity 0.83)
- Contributing Dimension Structure of Deep Feature for Coreset Selection (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.