Research map / Label-efficient and robust learning / Recognition and long-tail
GIO: Gradient Information Optimization for Training Dataset Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GIO, a task-agnostic data selection method that greedily picks training examples by gradient descent on a relaxed, kNN-estimated KL divergence between selected and target distributions, and shows gains on machine translation, spelling correction, and FashionMNIST.
Problem
Selecting a high-quality subset of a large, variable-quality training pool that matches a small unlabeled target distribution, without labels and independent of task or domain.
Approach
Relax the intractable KL-divergence subset objective to a continuous space, use the gradient of a kNN KL estimator (averaged over all k so the gradient is nonzero) to find an optimal point, then add the nearest real training example; quantize with K-means for scale and stop when KL increases.
Key results
- On WMT14 EN-FR, a model trained on 40% of GIO-selected data outperforms the full-data model; at 25% and 50% initialization GIO exceeds full-model BLEU by +0.4 and +0.6.
- In spelling correction GIO selects 73% high-quality data versus 50% for random and 55-59% for baselines.
- On FashionMNIST with 15,000 of 56,300 training images, GIO reaches 92.2% accuracy versus 91.5% for random and 94.5% for full data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select WMT14 training examples with GIO rather than random sampling | Transformer Big · WMT14 EN-FR, all reported initializations | BLEU: 41.2 vs 40.0; 42.2 vs 41.4; 42.3 vs 41.7 | Helped |
| select WMT14 training examples with GIO rather than random sampling | Transformer Big · WMT14 EN-DE, all reported initializations | BLEU: 24.3 vs 24.0; 27.0 vs 26.7; 27.9 vs 27.3 | Mixed |
| use GIO-selected data instead of the full corpus | Transformer Big · WMT14 EN-FR WMT 14 | BLEU: +0.4 and +0.6 BLEU | Helped |
| replace MPNet-Base-V2 with MiniLM-L12-v1 (ablation) | GIO · WMT14 EN-FR and EN-DE | BLEU: 0.7% difference on average | No clear effect |
| reduce quantization from K=1500 to K=1000 (ablation) | GIO · WMT14 EN-FR and EN-DE | BLEU: 0.4% difference on average | No clear effect |
| increase quantization from K=1500 to K=3000 (ablation) | GIO · WMT14 EN-FR and EN-DE | BLEU: 0.4% difference on average | No clear effect |
| select synthetic spelling data with GIO rather than random sampling | spelling correction data selection · synthetic spelling correction data | % high quality: 50% -> 73% | Helped |
| select FashionMNIST examples with GIO rather than random sampling | ResNet50 · FashionMNIST test | accuracy: +0.7% | Helped |
Surprising findings
- Training on a GIO-selected subset of WMT14 EN-FR (about 40% of the data) outperforms training on the full corpus.
- Derivative trick reported as 80% faster than naive hill-climbing on the KL objective (257s vs 1369s).
Limitations
- Gradient descent on a non-convex objective may not find the global optimum; the local density assumption is not guaranteed.
- A narrow target X may make the selected model perform poorly on inputs outside X.
- Stopping and reset heuristics are chosen per task; the modified KL estimator does not give zero divergence for identical distributions.
Experimental setup
Datasets: WMT14 WMT08-WMT13 dev sets 1 Billion Word benchmark (synthetic noised) BEA grammar correction corpus (BEA-60K) FashionMNIST BEA4660
Benchmarks: WMT14 EN-FR WMT14 EN-DE BEA4660 FashionMNIST
Metrics: BLEU accuracy KL divergence estimate % high quality data wall-clock time
Baselines: Random subset BM25 retrieval Self-pruning Submodular mutual information Full dataset training
Builds on: kNN KL divergence estimator submodular optimization BM25 self-pruning
Closest papers by meaning
- Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws (ICLR 2025 · similarity 0.85)
- DsDm: Model-Aware Dataset Selection with Datamodels (ICML 2024 · similarity 0.85)
- Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks (NeurIPS 2025 · similarity 0.84)
- Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyond (ICML 2024 · similarity 0.84)
- GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks (ICML 2024 · similarity 0.83)
- SAND: One-Shot Feature Selection with Additive Noise Distortion (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.