Research map / Learning theory and algorithms / Optimal transport
TAROT: Targeted Data Selection via Optimal Transport
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAROT, a targeted data selection method that uses whitened gradient feature distances and greedy optimal-transport distance minimization to pick training subsets matching a target distribution.
Problem
Influence-based greedy heuristics for targeted data selection work on unimodal targets but fail for complex multimodal target distributions, due to dominant feature bias and non-additive influence.
Approach
Replace additive influence scoring with an optimal transport distance between selected and target data, using a whitened gradient feature distance (Cholesky whitening plus normalization) as the ground cost, with greedy nearest-neighbor selection and an OT-increase stopping rule that also sets the selection ratio.
Key results
- On GTA5->Cityscapes segmentation, TAROT reaches the highest mIoU with about 24% of GTA5, while full GTA5 training lowers mIoU from 58.98 to 58.21.
- On instruction tuning, TAROT-OTM selects under 0.5% of the candidate data and matches or beats 5% LESS on MMLU and BBH (e.g., Qwen-2.5-7B BBH 68.9 vs 66.3 for LESS).
- Wayformer trained on TAROT-selected data ranks first on the nuScenes leaderboard with minADE5 0.92, ahead of Wayformer-All at 0.94.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 5% TAROT selection instead of 5% LESS | Llama-3.1-8B · MMLU | accuracy: 65.7 -> 66.0 | Helped |
| Use 5% TAROT selection instead of 5% LESS | Llama-3.1-8B · BBH | Exact Match: 62.6 -> 65.0 | Helped |
| Use 5% TAROT selection instead of 5% LESS | Qwen-2.5-7B · BBH | Exact Match: 66.3 -> 66.9 | Helped |
| Use 5% TAROT selection instead of 5% LESS | Qwen-2.5-7B · MMLU | accuracy: 74.3 -> 74.1 | Hurt |
| Use TAROT-OTM instead of training on all data | Qwen-2.5-7B · BBH | Exact Match: 63.8 -> 68.9 | Helped |
| Use TAROT-OTM instead of training on all data | Llama-3.1-8B · MMLU | accuracy: 63.8 -> 65.7 | Helped |
| Transfer TAROT-OTM-selected data from Llama-3.1-8B to Qwen-2.5-7B fine-tuning | Qwen-2.5-7B · BBH | Exact Match: 63.8 -> 68.7 | Helped |
| Train on TAROT-selected data instead of the full candidate dataset | Wayformer · nuScenes leaderboard | minADE5: 0.94 -> 0.92 | Helped |
| Add whitening to gradient distance estimation (ablation) | WGD · nuScenes | LDS: 0.007 -> 0.028; 0.010 -> 0.065; 0.017 -> 0.098 | Helped |
| Add gradient normalization (ablation) | WGD · nuScenes | LDS: 0.024 -> 0.028; 0.060 -> 0.065; 0.097 -> 0.098 | Helped |
| Add whitening and gradient normalization (ablation) | WGD · nuScenes | LDS: 0.005 -> 0.028; 0.008 -> 0.065; 0.015 -> 0.098 | Helped |
| Increase the OTM fold count (ablation) | TAROT-OTM · Motion prediction, test set and target set | model performance | Mixed |
Surprising findings
- Training on the full synthetic GTA5 dataset slightly degrades Cityscapes mIoU compared with the baseline, showing misaligned synthetic data can hurt.
- Smaller domain-relevant selected subsets often outperform models trained on much larger, less relevant datasets.
Limitations
- Can overfit to small or highly specific target datasets, compromising generalization.
- Does not explicitly encourage diversity in the selection objective.
Experimental setup
Datasets: GTA5 Cityscapes nuScenes Waymo Open Motion Dataset Argoverse 2 nuPlan CIFAR-10 Flan V2 CoT Dolly Open Assistant 1 MMLU
Benchmarks: Cityscapes nuScenes MMLU BBH CIFAR-10
Metrics: mIoU Brier-minFDE minADE5 LDS Accuracy Exact Match
Baselines: LESS DsDm TSDS TRAK Random selection
Builds on: TRAK LESS Sinkhorn algorithm UniTraj
Closest papers by meaning
- A CLIP-Powered Framework for Robust and Generalizable Data Selection (ICLR 2025 · similarity 0.84)
- TAROT: Towards Essentially Domain-Invariant Robustness with Theoretical Justification (CVPR 2025 · similarity 0.83)
- PEAKS: Selecting Key Training Examples Incrementally via Prediction Error Anchored by Kernel Similarity (ICML 2025 · similarity 0.82)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.81)
- Computational Budget Should Be Considered in Data Selection (NeurIPS 2025 · similarity 0.81)
- When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration (ICML 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.